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ABSTRACT 



A process allows an image capturing apparatus to be inte- 
grated with a personal computer to continuously display a 
video image of the imaging apparatus. Upon proper input by 
a user, or automatically after a timed interval, a snapshot of 
the video image is captured. An autodiscrimination process 
of the captured video image automatically decodes any 
bar-coded information present in the captured image and 
outputs the information. 

25 Claims, 6 Drawing Sheets 
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DECODING OF REAL TIME VIDEO tured into a named file saved by the computer, converted to 

IMAGING disk or otherwise. The user then must separately load a bar 

code decoding program such as the 1D/2D program, 

This application claims benefit of provisional application described above, into the system and load the disk separately 

60/030,360, filed Nov. 5, 1996. 5 as a file for execution by the bar decoding program. 

HELD OF THE INVENTION °^ P'°"*°!' ^'l!" '^j^ 1>P'°f 

if the image is not properly resolved by the digital camera, 

This invention relates to image capturing apparatus, and or other imaging device, then the stored image can not be 

more particularly to a method of capturing and decoding bar decoded properly. This means that the user must separately 

code information in real time from a continuously displayed ^'^ reaim the imaging device, download a new stored image, 

video signal of a particular target. rename the file, reload the bar code decoding program, and 

reload the newly stored image as input into the decoding 

BACKGROUND OF THE INVENTION program. Several such iterations might be needed, with each 

Image capture devices are known in the prior art for iteration taking a considerable amount of time, and produc- 

allowing diagnostic inspections to be performed, such as for frustration and inconvenience for the user, 

surgical or other medical procedures with minimum human Furthermore, this situation exacerbates if a target of 

intervention. Such devices include video output for allowing interest includes widely scattered symbols that are spaced by 

real time images of a target of interest to be viewed. more than a single field of view, because multiple decoding 

Examples of such devices used for medical purposes are operations would be required. As noted, each decoding 

described in U.S. Pat. Nos. 4,755,873, and 4,651,202, ^° operation would require a separate capture, loading and 

among others, which allow the image to be continuously decoding sequence, as described above. Because of the time 

viewed on a video monitor. Similar devices, such as required to perform the capture and decoding steps as 

borescopes, are used for inspection of steam vessels, auto- presently known, efi&dency and practicality are each limited, 
motive engines, and other applications extending into the 

military, industrial and scientific fields. ^ SUMMARY OF THE INVENTION 

In addition, bar code readers are also known for reading It is a primary object of the present invention to overcome 

ID and 2D bar code symbols, such as bar coded information the shortcomings of the prior art. 

in supermarkets etc. A variety of different bar code symbols ^ ^^-^^^ object of the present invention to 

are now known, for the ID barcode symbologies a number 3^ ^ovide a process which utilizes real-time video obtained 

of them have been developed to allow the encoding of larger ^^m a digital camera, endoscope, or other suitable image 

amounts of data, mcludmg Code 49, as described in U.S. capturing device for continuous display and integrates the 

Pat. No 4,794 239. ^sued to Allais, and PDF 417, as ^ideo capability in a computer to assess and decode any bar 

described m U.S.. Pat. No. 5 340.786, issued to Paviudus, et ^ode readable information found in the video and display the 

al. In these patents, stacked symbols partition the encoded 3^ encoded message, whQe simultaneously displaying the 

data into multiple rows, each including a respective ID bar yid^o image 

code pattern, all or most of which must be scanned and ■ t. . r,, 

decoded, then linked together to form a complete message. \' """r^^^! primary object of the present invention to 

Two dimensional (2D) matrix symbologies, have also been a physician, or o her pracutioner with an opportu- 

developed which offer orientation-free scanning and greater presentaUon of the video image or more 

data densities and capacities than their ID counterparts. 2D '° I'^^^'f^ '"^ Ti"' ? T^f ^ ^ ' 

matrix codes encode data as dark or light data elements ^'f readable symbol or symbols into the field of view 

within a regular polygonal matrix, accompanied by graphi- ^^^'^ f ^^.^^^"^ ^''^'^V'' ^P^^^^^^^ 

cal finder, orientation and reference stniciures. ^P^°^^ download the scannmg and decodmg programs. 

Bar code readers are known which discriminate between 45 ^} P™^^ "V^'^ '"T^""" t 

the different types of symbologies of each of the above ID "^^"^^ ^."P^^*^^ ^^^^^ ^""^^ ^ '^^'"^ ^""^^^y^ 

and 2D types. For example, optical readers capable of ID archival purposes. 

autodiscrimination are well known in the art. An early another primary object of the present invention to 

example of such a reader is the SCANTEAM™ 3OOO, allow a vanety of vanous imaging devices of varying types, 

manufactured by Welch Allyn, Inc. 50 ^^^^ ^ endoscopes, borescopes, and literally any digital 

Optical readers which are capable of 1D/2D discrimina- ^^P^|^ T'^-.f '''l!'' '""^^^ ' V° 

tion are less well known in the art, in that 2D symbologies "^terchangeably used with the above, irrespective of the 

are relatively new innovations. An example of a hand-held ^^P^' of imaging optics or lUummaUon systems used therein, 

reader having such capability is described in copending, another prunary object of the present invention to 

commonly assigned U.S. Ser. No. 08/504,643. A stationary 55 P^^""^^^ ^ ^^^^^^ "^^^^^ automatically and without 

2D image sensor is described in copending, commonly intervention capture and decode a video input while 

assigned U.S. Ser. No. 08/516,185, each of which is hereby allowing the real time image to remain displayed, 

incorporated by reference in their entirety. Most recently, a Therefore, and according to a preferred aspect of the 

barcode reader which performs 1D/2D autodiscrimination of present invention a process is provided for capturing and 

a target having multiple symbols is described in copending 60 ^^^coding bar-code information in real-time from a continu- 

and commonly assigned U.S. Ser. No. 08/697,914, filed Sep. ously displayed video signal, comprising the steps of: 

3, 1996. The reader captures a field of view and autodis- aiming an imaging apparatus at a target of interest, said 

criminates between the symbols within the captured field of target having optically readable and machine readable 

view. information contained thereupon; 

Typically, if a video image obtained with an image sensor 65 continually displaying a real time image of said target 

and displayed onto a computer monitor contains bar-coded from said imaging apparatus on a computer video 

information, the following occurs: First, the image is cap- monitor; 
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selectively capturing an image into the memory of said application, an electronic endoscope 60, such as described in 

computer; the '202 reference is shown in FIG. 3. 

decoding said image ifsaid bar-code readable inform alio Q The imaging device 14 also includes electronics which 

is contained on said real-time image while maintaining allow interfacing with a video capture card, also referred to 
said image on said display; and 5 as a frame grabber 18, which interfaces directly with a 

outputting the decoded information. computer 20. According to this embodiment, the computer 

According to a preferred embodiment, a captured image 20 is a Compaq Pentium 120 based PC and the frame 

with or without bars^ode data can be saved such as for grabber 18 is a Flashpoint Lite manufacmred by Integral 

archival purposes. Technologies, Inc. These items are each well known in the 

This application deals with integrating software which and require no further detailed discussion, 

allows a personal computer (PC) or other peripheral device The computer includes a microprocessor 22 which is a 

to be linked with literally any form of image capturing programmable control device which is able to receive, 

apparatus, such as an endoscope or other medical or other output and process data in accordance with a stored program 

diagnostic instrument having a solid-state imaging device. A maintained within either or both of a read/write random 

video image from the imaging device is displayed on the PC access memory (RAM) 24 and a hard drive 26. The 

monitor while the system automatically captures the image, described embodiment preferably refers to a computer hav- 

scans the image, and decodes and displays an encoded ing the Microsoft Windows operating system contained 

message, if such information is present. therein, though other similar systems can be utilized. 

The device is most useful for physicians or others who use The RAM 24 and the hard drive 26 are both connected to 

endoscopic or other diagnostic instruments having video a common bus 28 through which program data, including 

capabihty. The present process allows a real-time video address data, may be received and transmitted in either 

image to be repeatedly and selectively captured to allow direction to any circuitry that is also connected thereto, 

automatic decoding and outputting of encoded information. Included along the common bus 28 are user inputs, such 

Furthermore, the user can conveniently aim, capture and as a keyboard/ mouse arrangement 34 as weU as a video 

decode m a single operation. monitor 30 used to output the video signal, as is described 

Other objects, features, and advantages will become in greater detail below 

apparent form the following Detailed Description of the ^he overall operation of the system illustrated in FIG. 1, 

InvenUon, when read m combmation with the accompanying ^ow be described with reference to the main program 

drawmgs. represented by the flow chart of FIG. 2. 

BRIEF DESCRIPTION OF THE DRAWINGS Referring to FIG. 2, the main program begins with block 

105 which includes the activation of the program instruc- 

FIG. 1 is a block diagram of a preferred embodiment of tions which can be accomplished by calling up the program 

the present invention; from the hard drive 26 and loading the program into the 

FIG. 2 is a flow chart of the major process using a bar code 35 system using the standard Windows commands from cither 

reader and in accordance with a preferred aspect of the the file manager or otherwise as is known. Upon loading of 

present invention; the program, a set of system parameters defining initial sizes 

FIG, 3 is a perspective view of one embodiment of a bar ^^^^ window for the framegrabber 18, and other 

code reader which is used with the present invention; software variables, such as the mode of decoding, of 

uin ^ Co . fl« u t p u J- 4 f ,u nr^ ^0 triggering, etc are each activated. Each of the above param- 

FIG. 4 is a now chart of one embodiment of the 2D ^ u a - * a * u ^ 

f ,u * A - r J- * *u eters are descnbed in greater detail in the attached source 

portion of the autodiscnmmation process accordmg to the , u j • 1: . a j- t l j u . • 

invpntinn- attached as microfiche Appendix I, attached hereto, m 

^^^^^^^^^^ ^.^jj 37 Qjj^ 

FIG. 5 IS a flow chart of one embodiment of the ID ^^e initial start-up mode, a particular user input is 

portion of autodiscnmination program of the invention; ^^j^^^^^ ^ ^^^^^ -^^^^ ^^p^^^ ^^^^ 

FIGS. 6 through 9 show representative bar code symbols mechanism. According to this embodiment, a designated 

of types which may be decoded by the present invention; keyboard key, referred to as a "Hotkey", such as FIO, is 

FIGS. 10 through 14 are drawings which aid in the preset as the trigger. Alternately, the user can be prompted as 

understanding of the flow chart of FIG, 4; described in greater detail according to the source code, see 
nFTAF! FD nF^rRTPnoM OF THF ^° microfiche Appendix I, to utilize other keys on the keyboard 

DblAILbD DE^Ri™ OF THE ^r the mouse button as the trigger. Another alternate mode 

HON ^jjijjjj ^ snapshot can be taken automatically 

Referring to FIG. 1, there is shown a block diagram by the processor 22 after a predetermined time interval has 

according to a preferred embodiment of the present inven- elapsed, e.g. 2 seconds. Similarly, the initial startup mode 
tion. An imaging assembly 10 is provided for receiving an 55 provides for outputting of any decoded messages to the 

image of an object T and generating an electrical output monitor 30 after all of the symbols found have been 

signal indicative of the data optically encoded thereon, if decoded, such as adjacent the video image in a Windows 

any. The imaging assembly 10 may include an image sensor message box. Alternate outputting modes, however, are 

14, such as a ID or 2D CCD or CMOS solid state image contemplated allowing for several options for the user, such 
sensor together with an imaging optics assembly 16 for 60 as saving to a ASCII file, sending the messages to a host 

receiving and focusing an image of the object T onto a processor, a keyboard buffer, or other convenient means, 

substrate of the image sensor 14. Particular assemblies of The imaging assembly 10, using the imaging optics 16 

this type are described in U.S. Pat. Nos. 4,755,873, and focusses an image of the target T onto the image sensor 14 

4,651,202, each of which are incorporated by reference. It as is commonly known. The image of the target is then 
will be readily apparent from the description of the 65 converted into analog electrical signals which are transmit- 

invention, however, that literally any device having an ted to the frame grabber 18 having circuitry to perform the 

internal image sensor is applicable. For purposes of this analog to digital conversion to provide an anray of pixels as 
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defined by default parameters which is displayed as a direction will be readily detectable in a second direction 

continuous video signal in the monitor 30 per block 120. perpendicular to the first direction. 

Details relating to the above are generally known to those of Preferably, an activity profile of the stored image data is 

skill in the field, and require no further elaboration herein. constructed on the basis of only a selected, relatively sniall 

The frame grabber 18 is preferably supplied as an input card s number of image data elements that are distributed across 

to the computer 20 and serves as the video driver, though the the field of view that corresponds to the stored image data, 

frame grabber 18 can also be more than one driver, also as Using a relatively small number of data elements is desirable 

is commonly known. to increase the speed at which the symbol can be imaged. 

Referring to the flow chart of FIG. 2, and after the video ^^^^^^^ P^^^^ ^^^cted as the points which lie 

image of the target has been supplied by the image sensor to lo ^he intersections of an X-Y samp mg ^id such as that 

.u -.in ui 1 nn j shown m FIG. 10. The Spacing of the luies defining this grid 

the monitor 30perblockl20,theuserperblockl25.sready ^^tical to the present invention, but does affect the 

to capture an instantaneous image (hereinafter referred to as resolution with which the activity profile of the image can be 

a snapshot") for storage into RAM 24 per block 130 and for measured 

attempting to decode any symbols present in the fiekl of when the processor 22 has determined the activities of the 

view per block 135. As noted above, the initial mode 15 .elected data points, it is directed to block 315, which causes 

contemplates the user must elect the capture of a snapshot jt ^ i^ok for candidate bar code symbols by identifying 

using the Hotkey, FIO, to execute block 125, though as also ^^^^^ „f ^gj, ^^^jjy ^hi^ conveniently done by 

noted above, a number of alternate modes are available by determining those sets of image data points having activities 

which the user or the processor 22 may capture a snapshot. ^^ich exceed a predetermined threshold value. Asimplifled, 

Regardless of the mode selected, the user aims the image ^ one-dimensional representation of this step is illustrated in 

sensor 16 at the object T until a resolved image is displayed piG. 11, wherein those image data points having an activity 

m the monitor 30. Upon depression of the "hotkey", the ,hat exceed a threshold value TO are labeled as a candidate 

processor 22 proceeds automatically to blocks 130 and 135, symbol region CSRl 

which call for the capture and attempt to decode an instan- in embodiments which are adapted to find and decode all 

aaeous image oi^ ine signai.^ reteried to herealter _as a ^= .^^ ^^^^ 

occur in fields of view that include a 

snapshot , referring to an instantaneous digital signal i^,,iit/of bar code symbols, the result is the identification 
which IS stored mto RAM. In passing, it should be noted that ^ j^^^j^ ^^^^^^^ ^ ^ol regions (CSRs), any one 
the acquisition of the snapshot of the video image does not ^^^^ ^^^^^ ^ ^ar code symbol. Whether or not 
impact the video image which remams displayed on the ^^^^ ^ ^^^^^ determined on the basis of 
monitor, as a real time image. 30 ^^^^^^^ decodable. According to this embodiment, 
Image capture of the snapshot to RAM 24 automaticaUy the processor is instructed to select one of the CSRs accord- 
causes the processor 22 to attempt to decode the snapshot as jug to a suitable selection rule, such as the largest CSR first, 
shown in the blocks 130, 135 in FIG. 2. This decoding the CSR nearest the center of the field of view first, the CSR 
involves a discrimination process, depending on whether with the highest total activity first, etc., and then attempt to 
any ID and or 2D symbols are present in the field of view. decode each of the symbols, to attempt to decode the first 
This autodiscrimination process is described in greater detail symbol and stop, depending on whether or not the symbol 
below. If the decoding is successful, per the decision block has been decoded. 

140, the messages are outputted per block 145. If the attempt Once all of the CSRs have been located, the processor 22 

to decode the messages is unsuccessful, the user may opt to ^ directed to block 320 which, calls for the processor to 

recapture the video signal per block 125, and block 145 is ^.j^^t t^e largest (or most centrally located) as yet unexam- 

bypassed. The user may aUernately elect to store the cap- cSR for further processing, and then to proceed to 

tured image per block 150, into the hard disk 26. ^^^^^ 325, letter block then causes the processor 22 to 

Generally speaking, the flow chart of FIG. 4 illustrates the find the centroid or the center of gravity of that CSR, before 

ID portion of a complete 1D/2D autodiscrimination process, proceeding to block 330. An example of such a centroid is 

while FIG. 5 illustrates the 2D portion of the complete labeled C in FIG. 12. Because the steps involved in finding 

1D/2D autodiscrimination process. a centroid or center of gravity are well known, they will not 

Tuming fust to the flow chart of FIG. 4, there is shown the be described in any detail herein. 

ID portion of the autodiscrimination process, which oper- On encountering block 330, the processor 22 is directed 

ates on the image data stored in RAM 24. The image data 50 to examine the selected CSR by defining exploratory scan 

will comprise a gray scale representation of the 2D image lines therethrough, determining the activity profile of the 

formed on the image sensor. CSR along those scan lines, and selecting the scan line 

On encountering block 305, the processor 22 is directed having the highest total activity. In the case of a ID bar code 

to calculate the "activities" of selected image data elements. symbol, this will be the direction most nearly perpendicular 

The "activity" of a point P as used herein comprises a 55 *o the direction most nearly perpendicular to the direction of 

raeasureof the rate of change of the image data over a small the bars, i.e., the optimum reading direction for a ID 

two dimensional portion of the region surrounding point R symbol. 

This activity is preferably calculated along any two arbi- On exiting block 330, the processor 22 encounters blocks 

trarily selected directions which are mutually perpendicular 335 and 340. The first of these blocks 335 .scans a scan line 

to one another, as shown by the lines parallel to directions 60 counter to zero; the second block 340 defines an initial 

X and Y of FIG, 10. One example of an activity calculation working scan line through the centroid in the previously 

is that which is based on the squares of the gray scale determined direction of highest activity. The result of this 

differences of two pairs of points P1X-P2X and P1Y-P2Y operation is the definition, in the image data space repre- 

that are centered on point P, as shown in FIG. 10. Two sentation of the CSR, of a working scan Une, such as SC«^0 

mutually independent directions are used because the ori- 65 in FIG. 12. 

entatioD of the symbol is unknown and because a high Once the initial scan line has been defined, the processor 

activity level that by chance is difficult to detect in a first 22 is directed to block 345 to calculate, by interpolation 
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from the image data of the CSR, the values of the sampling 
points that lie along this scan line. This means that, for each 
sampling point on the initial scan line, the processor 22 will 
calculate what brightness the sampling poim would have if 
its brightness were calculated on the basis of the weighted 
brightness contributions of the four nearest brightness points 
of the CSR. These contributions are illustrated by the dotted 
lines which join the sample point SP of FIG. 13 to the four 
nearest image data points DPA-DPD. So long as these 
sampling points are more closely spaced than the image data 
points, this interpolation procedure will be performed on a 
subpixel basis, and this will produce a usably accurate 
representation of the image data along the scan line. The 
result of the subpixel interpolation of the sampling points on 
a representative scan line of this type is shown in FIG. 14. 
Because the particulars of the subpixel interpolation process 
are known to those skilled in the art, this process will not be 
further described herein. 

Once the above-described scan line data has been 
calculated, the processor 22 is directed to block 350, which 
calls for it to binarize the scan line data, i.e., convert it to a 
two-state representation of the data which can be processed 
as a candidate for ID decoding. One such representation is 
commonly known as a timer count representation. One 
particularly advantageous procedure for accomplishing this 
binarization process is disclosed in U.S. Pat. No. 5,586,960, 
which is hereby incorporated by reference. 

On exiting block 350, the processor 22 will be in posses- 
sion of a potentially decodable two-state ID representation 
of the CSR. It then attempts to decode this representation, as 
called for by block 355. This attempted decoding will 
comprise the trial application to the representation of one ID 
decoding program after another until the latter is either 
decoded or determined to be undecodable. Because decod- 
ing procedures of the latter type arc known to those skilled 
in the art, they will not be discussed in any further detail. 

As the ID autodiscrimination process is completed, the 
processor 22 is directed to decision block 360 which causes 
it to continue along one of two different paths, depending on 
whether or not decoding was successful. If the decoding was 
not successful, the processor 22 will be caused to loop back 
to block 340, via blocks 365 and 370, where it will be caused 
to generate a new scan line SC=0, but that passes either 
above or below centroid C. This looping back step may be 
repeated many times, depending on the "spacing" of the new 
scan lines, until the entire CSR has been examined for 
decodable ID data. If the entire CSR has been scanned and 
there has been no successful decode, the processor 22 is 
directed to exit the above -described loop via block 375. As 
used herein, the term "paraUel" is broadly used to refer to 
scan lines or paths which are similarly distorted (e.g. 
curvilinear) as a result of foreshortening effects or as a result 
of being imaged from a non-planar surface. Since compen- 
sating for such distorting effects is known, as indicated, for 
example, by U.S. Pat. No. 5,396,054, it will not be discussed 
in further detail. 

Block 375 serves to direct the processor 22 back to block 
320 to repeat the above -described selection, scanning and 
binarizing steps for the next unexamined CSR, if one is 
present. If another CSR is not present, or if the processor's 
program calls for an attempt to decode only one CSR, block 
375 causes the processor 22 to exit the flow chart of FIG. 4 
to begin an attempt to decode the then current set of image 
data as a 2D symbol, in accordance with the flow chart of 
FIG. 5. If other CSRs are present, block 375 directs the 
processor 22 back to block 320 to repeat the selection, 
scanning and binarizing process to the next CSR, and the 
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next, and so on, until there is either a successful decode 
(block 360) or all of the CSRs have been examined (block 
375). 

If the processing of the first CSR has resulted in a 

5 successful decode, block 360 directs the processor 22 to 
block 380, which causes it to determine whether the decoded 
data indicates that the CSR contains a ID stacked symbol, 
such as a PDF417 symbol One example of such a symbol 
is shown in FIG. 9. If it is not, i.e., if the decoded symbol 

10 includes only a single row of bars, the ID data is stored for 
later outputting in accordance with block 145 of the main 
program of FIG. 2, as called for by block 385. Alternately, 
the data may be output immediately and block 145 later 
skipped over. Then, if there are no remaining unexamined 

15 CSRs, the processor is directed to exit the flow chart of FIG. 
4 via block 390. If, however, there are CSRs remaining, 
block 390 will direct the processor back to block 320 to 
begin processing the next CSR, and the next, and so on until 
all CSRs have been examined and decoded (block 390) or 

20 examined and found to be undecodable (block 375). 

If, on encountering block 380, the decoded data indicates 
that the CSR contains a ID stacked symbol, the above- 
described processing is modified by providing for the rep- 
etition of the scanning-digitizing process, beginning with 
block 340. This is accomplished by blocks 395, 396, and 397 
in a manner that will be apparent to those skilled in the art. 
Significantly, by beginning the repeating of the process at 
block 340, all additional scan lines defined via the latter path 
will be parallel to the first decodable scan line, as required 
by a ID stacked symbol, at least in the broad sense discussed 
earlier. 

In view of the foregoing, it will be seen that, depending 
on the number of CSRs present, the flow chart will cause all 
ID symbols in the image data to either be decoded or found 
to be undecodable before directing the processor 22 to exit 
the same. 

The 2D autodiscrimination flow chart of FIG. 5 may be 
processed after the processing of the ID autodiscrimination 
flow chart of FIG. 4 has been completed. It may also be 
processed without the flow chart of FIG. 4 having been 
processed, i.e., the ID portion of the 1D/2D autodiscrimi- 
nation process may be skipped or by passed. (In principle, 
the steps of the 2D portion of the 1D/2D discrimination 
process (FIG. 5) may also be processed before the ID 
portion thereof (FIG. 4). Such arrangements are well within 
the intended scope of the present invention. 

Referring to FIG. 5, there is shown a flow chart of the 2D 
portion of the 1D/2D autodiscrimination process. When the 

50 flow chart of FIG, 5 is entered, the image data that is stored 
in RAM 24 is the same as that which would be stored therein 
if the flow chart of FIG. 4 were being entered. This data will 
comprise an array of 8-bit gray scale image data elements 
produced by the image sensor 16 and its associated signal 

55 and processing and AID converter circuits contained in the 
framegrabber 18. 

The flow chart of FIG. 5 begins with a block 505, which 
directs the processor 22 to convert the gray image data 
representation stored in RAM 24 into a two-state or bina- 

60 rized representation of the same data. ITiis may be accom- 
plished in generally the same manner described earlier in 
connection with FIG. 11, i.e., by comparing these gray scale 
values to a threshold value and categorizing these values as 
Is or Os, depending on whether they exceed that threshold 

65 value. 

Once the image data has been binarized, the processor 22 
continues on to block 510, which causes it to identify and 
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locate all of the 2D finder patterns that appear in the field of 
view of the image data. This is preferably accomplished by 
examining all of the candidate 2D finder patterns (CFPs) that 
are present and identifying them by type, i.e., identifying 
whether or not they are bulls eye type finder patterns, 
waistband type finder patterns or peripheral type finder 
patterns. An example of a bulls eye type finder pattern is 
shown in the central portion of the 2D bar code symbol of 
FIG. 6, which symbol encodes data in accordance with a 2D 
matrix symbology named "Aztec". An example of a waist- 
band type finder pattern is shown in the middle portion of the 
2D bar code symbol of FIG. 7, which symbol encodes data 
in accordance with a 2D matrix symbology named "Code 
One", An example of a peripheral type finder pattern is 
shown in the left and lower edges of the 2D bar code symbol 
of FIG. 8, which symbol encodes data in accordance with a 
2D matrix symbology known as "Data Matrix". The finder 
identification processing is performed by applying to each 
CFP, in turn, a series of finder pattern finding algorithms of 
the type associated with each of the major types of finder 
patterns. Since such finder finding algorithms are known for 
finders of the waistband and peripheral types, these algo- 
rithms will not be discussed in further detail. One example 
of a finder finding algorithm for a waistband type finder may 
be found, for example, in "Uniform Symbology Specifica- 
tion Code One", published by AIMAJ.S.A. Technology 
Group. Finder finding algorithms for bulls eye type finders 
that include concentric rings, (e.g. MaxiCode) are also 
known and therefore will also not be described in further 
detail. 

Particularly advantageous for purposes of this 
embodiment, is a bulls eye finder finding algorithm of the 
type that may be used both with 2D symbologies, such as 
MaxiCode, that have bulls eye finder patterns that include 
concentric rings and with 2D symbologies, such as Aztec, 
that have bulls eye finder pattems which include concentric 
polygons. A finder finding algorithm of the latter type is 
described in described in copending and commonly assigned 
U.S. Sen No. 08/504,643, which has been incorporated 
herein by reference. The Aztec 2D bar code symbology itself 
is fully described in U.S. Ser. No. 08/441,446, which has 
also been incorporated by reference. 

Once all of the finder pattems have been located and their 
types have been determined, the processor 22 is directed to 
decision block 515. This block affords the processor an 
opportunity to exit the flow chart of FIG. 5, via exit block 
545, if no 2D finder patterns could be found and typed. This 
block speeds up the execution of the program by skipping 
over decoding operations which have no hope of success 
without their associated finder pattern. 

If a finder pattern has been found and typed, the processor 
22 is directed to block 525. This block causes the processor 
22 to select for decoding the bar code symbol whose finder 
is closest to the center of field of view of the image data. 
Optionally, the processor may be instructed to find the 
largest 2D bar code symbol that uses a particular 2D 
symbology or the 2D bar code symbol using a particular 2D 
symbology which is closest to the center of the field of view 
of the image data. Once this selection is made, the processor 
attempts to decode that symbol, as called for block 530. If 
this decoding attempt is successful, as determined by deci- 
sion block 535, the resulting data may be stored for output- 
ting in accordance with block 135 of the main program of 
FIG. 2, as called for by block 540. Alternately, the decoded 
data may be outputted immediately and block 135 later 
skipped over. If the decoding attempt is not successful, 
however, block 540 is skipped, and the processor 22 is 
directed to decision block 545. 
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Block 545 will direct the processor 22 back to block 525 
to process the next 2D symbol, i.e., the symbol whose CFR 
is next closest to the center of the field of view. The 
above-described attempted decoding and storing (or 
s outputting) steps will then be repeated, one CFR after 
another, until there are no more symbols which have usable 
finder data pattems. Finally, when all symbols having usable 
finder pattems have either been decoded or found to be 
undecodable, the processor will exit the flow chart of FIG. 
5, via block 550, to return to the main program of FIG. 2. 

All of the decoded messages are then outputted to the 
monitor 30, and the entire output is displayed. According to 
this embodiment, the contents of each of the bar code 
messages are output to the WINDOWS message box, though 
other alternatives are possible. 

The invention has been described with reference to a 
specific embodiment, but it will be readily apparent that 
other similar embodiments can be utilized to provide decod- 
ing and outputting of a real time video signal and as covered 
by the accompanying claims. 
20 We claim: 

1. A process for capturing and decoding bar-code infor- 
mation in real time from a continuously displayed image 
video signal, comprising the steps of: 

aiming an imaging apparatus at a target of interest, said 
25 target having at least one of optically readable and bar 
coded information contained thereupon; 
continually displaying a real time image of said target 

from said imaging apparatus; 
selectively capturing and storing an instantaneous image 
30 of said target into the memory of a computer; 

determining if bar-coded information is present in said 
stored image; 

decoding bar-code information if bar-code readable infor- 
mation is contained on said instantaneous stored image 
35 while maintaining the display of said real-time image; 
and 

outputting the decoded bar-coded information. 

2. A process as recited in claim 1, wherein said selective 
capturing step is performed automatically after a predeter- 

40 mined time interval. 

3. A process as recited in claim 1, wherein said capturing 
step is selectively performed by input supplied through input 
means finked to said imaging apparatus. 

4. A process as recited in claim 3, wherein said input 
means is a keyboard. 

5. A process as recited in claim 3, wherein said input 
means is a mouse. 

6. A process as recited in claim 3, wherein said input 
means is a signal from one of at least one of an external, 
remote and host device. 

^0 7, A process as recited in claim 1, including the step of 

outputting said decoded information adjacent said real-time 

displayed image. 
8. A process as recited in claim 1, including the step of 

outputting said decoded information to a computer file. 
55 9. A process as recited in claim 1, including the step of 

outputting said decoded information to a computer keyboard 

buffer. 

10. A process as recited in claim 1, including the step of 
outputting wherein said decoded information to a pop-up 

60 dialogue box. 

11. A process as recited in claim 1, including the step of 
outputting said decoded information to an application's 
cUpboard. 

12. A process as recited in claim 1, including the step of 
65 integrating said process with an external software applica- 

lion and outputting said decoded information to said appU- 
cation. 
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13. A process as recited ia claim 1, including the step of 
storing captured images in which at least some of said 
images include bar code information. 

14. A process for capturing and decoding bar-coded 
information in real time from a continuously displayed video 
signal of a target of interest, said process comprising the 
steps of: 

aiming an imaging apparatus at a target of interest; 
continually displaying a real time image of said target 

obtained from said imaging apparatus; 
selectively capturing an instantaneous displayed image of 

said target and storing said image into computer 

memory; 

determining if bar-coded information is present in said 
captured and stored image; 

decoding all bar-coded information present in said cap- 
tured and stored image while continually displaying 
said real time image of said target; and 

outputting the decoded bar-code information onto said 
real time displayed image. 

15. Aprocess as recited in claim 14, wherein said selective 
capturing step further includes the step of automatically 
capturing said instantaneous image after a predetermined 
time interval. 

16. Aprocess as recited in claim 14, wherein said target 
contains an indeterminate number of bar^coded segments 
disposed indiscriminately throughout, said process includ- 
ing the further steps of: 

identifying all bar-coded information in the field of view 

of said captured image; 
decoding all bar-coded information segments found in 

said captured image; and 
outputting the decoded bar-coded information of each 

segment onto said real-time displayed image, wherein 

the bar-coded information can include distinct ID and 

2D symbologies. 

17. A process as recited in claim 14, wherein said cap- 
turing step includes the step of selectively inputting a 
capture signal to said imaging apparatus. 

18. Aprocess as recited in claim 14, including the step of 
outputting said decoded information to a computer file. 
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19. Aprocess as recited in claim 14, including the step of 
outputting said decoded information to a pop-up dialogue 
box which is displayed. 

20. Aprocess as recited in claim 14, including the step of 
5 additionally storing said captured images and decoded bar- 
coded information into memory after said outputting step. 

21. Aprocess as recited in claim 20, wherein said storing 
step includes the storing of images regardless of whether 
bar-coded information has been determined during said 

10 determining step. 

22. Apparatus for capturing and decoding bar-coded infor- 
mation from a target of interest comprising: 

imaging means for imaging a target of interest, said target 
having at least one of optically readable and bar-coded 
15 information contained therein; 

processing means for processing an imaged target; 

display means for continually displaying a real-time 
image of said target from said imaging and processing 
means; 

20 . 

image capture means for selectively capturing at least one 
image displayed by said display means; 

scanning means for scanning said at least one captured 
image and for determining the presence of bar-coded 
information in the field of view of said at least one 
captured image; 

decoding means for decoding any bar-coded information 
detected by said scanning means; and 

output means for outputting the decoded bar-coded infor- 
30 mation to said display means. 

23. Apparatus as recited in claim 22, including discrimi- 
nation means for discriminating the type of bar-coded infor- 
mation present in said at least one captured image. 

24. Apparatus as recited in claim 22, including input 
35 means connected to said image capture means for inputting 

a signal to operate said scanning means and said decoding 
means when an image is captured. 

25. Apparatus as recited in claim 22, including storage 
means for selectively storing images captured by said image 

40 capture means regardless of whether bar-coded information 
is present, 

4t Ht * ♦ * 
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Detailed Description Text - DETX (44) : 

When line conditions degrade, the analog front end 514 
will become aware of 

the degradation as a result of incoming frame synchronous 
signal 570. Analog 

front end 514 will then notify the digital signal processor 
modem 516 and host 

processor 218- Host processor 218 will then switch from a 
standard operation 

to an exception operation mode. Host processor 218 has 
three options to lower 

the bit rate in order to accommodate and correct the 
degradation. Option 1 

would be for the host processor 218 to notify the pixel 
processor 206 and 

select a coarser quantization level 572, Option 2 would be 
to drop the frame 

update rate and increase the interpolation rate 574. 
Option 3 would be to drop 

from GIF to QCIF 576. When the error processor 526 detects 
more than two 

single bit errors, the error processor 526 will notify the 
pixel processor 206 

and host processor 218. Host processor 218 again has two 
options . Either 

pixel processor 2 06 can request for an retransmission or 
host processor 218 can 

delete the complete macro block 477 and wait until the next 
macro block is 

sent. Meanwhile host processor 218 will send the old macro 
block 308 from the 

frame memory 214 and use it to update the display. 
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ABSTRACT 



The running real time length of combined video and audio 
signal programs is shortened or lengthened by deleting or 
repealing individual fields or frames and corresponding 
amounts of audio segments. The video and audio portions of 
the programming material are separated and subjected to 
processing through a pair of program time changing units. 
The video portion is processed by deleting individual fields 
or frames on a manual, periodic or automatic basis. Manual 
deletion is done by an operator observing the program 
material on a monitor. Periodic deletion is performed auto- 
matically after the operator specifies the total amount of time 
(or number of fields or frames) lo be deleted, with every ith 
frame or fi^eld deleted regardless of content. Automatic 
deletion is done in a fashion similar to periodic deletion, but 
the fields or frames are examined and are deleted on the 
basis of the amount of between frame motion. Audio seg- 
ment deletion is done either manually, periodically or 
automatically, and the audio segments removed need not 
match the deleted frame video, but may be taken from 
different frames, so long as the total time length of the 
deleted audio segments equals the total time of the deleted 
video frames, and also provided that the differential delay 
between the processed video and processed audio does not 
exceed the lip sync criterion. A pause function disables the 
deletion or insertion process for program materials which 
may not be altered in any way. 

6 Claims^, 2 Drawing Sheets 
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VIDEO/AUDIO PROCESSING SYSTEM 
PROVroiNG REAL TIME PROGRAM 
DURATION ALTERATION 

This is a divisional of application Sen No. 09/426,844, 
filed Oct. 26, 1999 which is a divisional of application Ser. 
No. 08/552,357, filed Nov. 2, 1995 now U.S. Pat. No. 
5,995,153. 

BACKGROUND OF THE INVENTION 

This invention relates to video signal processing methods 
and systems. More particularly, this invention relates to 
methods and systems for altering the content of video 
program material to expand or contract the total length of an 
entire program or program segment. 

\ldeo signal processing systems and methods are known 
for editing the content of an entire program or program 
segment in order to expand or contract the total program run 
time to match a desired run length or time segment. 
Frequently, a program or commercial which is scheduled for 
a predetermined broadcast time slot has a total running lime 
which does not match exactly the time slot. In such cases, it 
is necessary to edit the program in order to fill the time slot 
exactly. In known systems, the program material must first 
be recorded on some suitable recording medium, such as 
magnetic tape, after which portions of the video program are 
deleted or repeated in order to contract or expand the 
running time to match the time slot. Such systems suffer 
from the disadvantage that the program to be edited cannot 
be simultaneously broadcast, but must be time delayed by 
the recording process. In addition, this technique is incom- 
patible with live events, such as soccer matches, football 
games and the like, which must be broadcast and viewed 
substantially simultaneously. Efforts to date to provide real 
time video time editing to contract or expand the program 
length to match a desired run length have not met with 
success to dale. 

SUMMARY OF THE INVENTION 

The invention comprises a method and system for pro- 
viding real time video program expansion or contraction 
which relatively inexpensive to implement, easy to operate, 
and effective matching program run time with a predeter- 
mined run length. In addition, the invention is effective in 
creating surplus broadcast from any program in order to 
provide additional broadcast time for other information, 
such as commercial spots, public service announcements, 
and the Hke. 

From a processing standpoint, the invention comprises a 
method of adjusting the total time length of a program 
having a fixed time duration by deleting or repeating indi- 
vidual frames or fields of video and audio segments on-the- 
fly, either on a fixed periodic basis, an automatic basis or 
manually using an operator controlled deletion or insertion 
circuit. The audio segments may or may not correspond to 
the frames or fields, but need only match the total time value 
of the deleted or repeated video frames or fields. 

To contract a given program in the manual mode of 
operation, the operator monitors the video program material 
and deletes one frame or field at a time. A counter accumu- 
lates the time value of the sum of deleted frames or fields and 
displays this total to the operator. Once the desired amount 
of additional broadcast time has been accumulated, the 
original program material is permitted to be passed through 
unmodified. To expand a given program in the manual mode, 
the operator monitors the video program material and 
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repeats one frame or field at a time, and the time accumu- 
lation counter keeps track of the total amount of time value 
of the repeated frames or fields and displays this information 
to the operator. Once the correct amount of time has been 

5 added to the original program material, the original program 
material is permitted to pass through unmodified. During the 
video deletion or repetition, corresponding segments of 
audio are deleted or repeated. 
In the periodic mode of operation, the operator enters the 

10 total amount of time to be deleted or added to the original 
program material length, and frames or fields and corre- 
sponding audio segments are deleted or added automatically 
in a periodic manner, e.g. one frame every five frames, one 
field every five fields, etc. The time counter accumulates the 

15 running total of deleted or added time, and terminates the 
deletion or insertion process when the running total equals 
the preselected amount 

In the automatic mode of operation, the operator adds the 
total amount of time to be deleted or added to the original 
program material length, and frames or fields and audio 
segments arc deleted or added automatically, with the choice 
of particular frames/fields and audio segments being deter- 
mined by video motion detectors and audio pitch and level 
detectors, so that optimal deletion or insertion is effected. 

25 

In both the periodic and automatic modes of operation, an 
optional pause function protects any specially selected sen- 
sitive programming material from being affected by the time 
change processing. The pause function itself may be manu- 
ally controlled by an operator, or automatically performed 
by detecting special marker information inserted in prede- 
termined frames/fields, or in reserved portions of same (i.e. 
during vertical blanking). 
The invention enables on-the-fly adjustment of the run- 

35 ning time of program material without affecting the actual 
broadcast of the program material. In addition, for programs 
of indeterminate length, the invention enables a fixed 
amount of time to be added or deleted over a preselected 
time period. For example, the invention can be used to gain 

40 a preselected fixed amount of time — e.g. ten minutes over a 
one hour broadcast time period — using either the manual or 
periodic modes of operation. Most importantly, the time 
deletion or insertion does not visibly affect the program 
content as experienced by the viewer, so that the program 

45 material can be enjoyed to the same extent as the original 
program material. 

For a fuller understanding of the nature and advantages of 
the invention reference should be made to the ensuing 
detailed description of the invention, taken in conjunction 

50 smith the accompanying drawings. 

BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 is a block diagram of a video and audio processing 
system incorporating the invention; 

FIG. 2 is a schematic diagram illustrating the frame 
deletion process; and 

FIG. 3 is a schematic diagram illustrating the frame 
insertion process. 

DETAILED DESCRIPTION OF THE 
PREFERRED EMBODIMENTS 

Turning now to the drawings, FIG. 1 is a block diagram 
of a video and audio processing system incorporating the 
65 invention. As seen in this figure, the video and audio 
portions of standard program signals (e.g. NTSC, PAL, 
SECAM or the like) are initially separated into the indi- 
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vidual video and audio components using conventional 
circuitry (not shown). The video portion is applied to the 
input of a first signal processor unit 12 incorporating an 
analog to digital converter 14, a digital video memory 15, 
and a digital to analog converter 16. The video is first 
converted from analog to digital form in analog to digital 
converter 14 and stored in digital video memory 15. Digital 
video output from digital video memory 15 is converted 
back to analog form in digital to analog converter 16 and 
supplied to a first output terminal 17 as time changed video. 

Similarly, the audio portion of the programming material 
is supplied to an audio signal processing unit generally 
designated with reference numeral 22 and including an 
analog to digital converter 24, a digital audio memory 25, 
and a digital to analog converter 26. The audio portion of the 
programming material is thus converted from analog to 
digital form in analog to digital converter 24, stored in 
digital form in digital audio memory 25, and converted from 
digital to analog form in digital to analog converter 26. The 
analog output from digital to analog converter 26 is supplied 
to a second output terminal 27 as time charged audio. Analog 
to digital converters 14, 24 and digital to analog converter 
16, 26 are conventional units having an appropriate bit size 
(e.g. 8 bits per sample) and a clocking speed compatible with 
the frequency content of their respective input signals. For 
example, the clock rate for analog to digital converter 14 
used in the preferred embodiment is 13,5 MHz, although 
other clock frequencies may be employed so long as they 
comply with the known requirements of sampling theory. 
The bit size used in the preferred embodiment for analog to 
digital converter 24 is 16 bits per sample, and the clock rate 
employed is 48 KHz. Similar considerations apply to digital 
to analog converters 16 and 26. 

Digital video memory 15 is a conventional digital storage 
unit having a capacity at least equal to the maximum 
accumulation time expected to be afforded by the system. 35 
For example, for a system designed to accumulate 30 
seconds worth of frame time over a one-hour total interval, 
digital video memory 15 would have a capacity to store at 
least 900 frames of NTSC video. Similar considerations 
apply to digital audio memory 25: however, the total storage 40 
capacity of digital audio memory 25 may be substantially 
less than that of digital memory 15 due to the lower 
frequencies at which audio is conveyed. In the preferred 
embodiment, digital audio memory 25 has a storage capacity 
of 30 seconds (i.e., the same time storage capacity of digital 
video memory 15). 

Video processing unit 15 incorporates circuitry termed a 
A segment circuit 18 which may be manually overridden by 
a manual video control 19. The purpose of the A segment 
circuitry 18 is to either delete or insert frames of video from 
the sequence of frames stored in digital video memory 15 on 
a programmed basis. Frame deletion is done by simply 
skipping over a frame in the normal sequence of frames and 
is described below with reference to FIG. 2. Frame insertion 
is accomplished by simply repeating a given frame in the 
frame sequence and is described below with reference to 
FIG. 3. In the periodic mode of operation, the frame deletion 
or insertion rate is set by the manual control 19, which the 
operator uses to dial in the total number of frames or amount 
of time to be deleted or added to the original program 
content during the initial stage of the signal processing. 
Thus, for example, if the operator wishes to delete ten 
seconds of lime over a one-hour period, that number is 
entered by means of manual control 19 into the A segment 
circuit 18, and the circuitry periodically deletes every ith 
frame until a total of 10 seconds worth of frame time has 
been deleted or saved. 
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During the periodic frame deletion or insertion 
processing, a AV circuit 20 keeps track of the total time 
value of the deleted or inserted frames. AV circuit 20 
essentially comprises a counter which receives frame dele- 
tion or insertion signals from the segment circuit 18 and 
either increments or decrements in response to each deleted 
or inserted frame. 

Audio processing circuit 22 is provided with similar A 
segment circuitry 28 and A audio circuitry 30 for the similar 
purpose of deleting or adding audio segments and keeping 
track of the total time value of the segments deleted or 
inserted, A segment circuit 28 is controlled by manual 
control unit 19 in tandem with A segment circuit 18. 
However, the actual audio portions which are deleted or 
repeated need not correspond exactly to the same frames 
which are deleted or repeated by video processing unit 12: 
in fact, the audio portions which are deleted or inserted may 
be segments of audio signals from different frames. It is 
STifficient that any time delay between the video and the 
audio signal portions subjected to the time variation pro- 
cessing and presented to output terminals 17, 27 not exceed 
±3 video frames, with a maximum difference of tl frame 
being preferred. By observing these constraints, no lip sync 
error is introduced to the original programming material in 
an observable fashion. 

As indicated by the legends GL and External Sync Gen, 
the video processing unit 12 may be coupled to and driven 
by an external sync generator, such as a studio sync 
generator, so that the video processing can be done in 
synchronous fashion with other video broadcasting or repro- 
duction equipment. 

FIG. 2 is a schematic diagram illustrating the frame 
deletion process in either the periodic or manual mode. As 
schematically represented in this figure, the leftmost column 
represents the sequence of video frames incoming on input 
terminal 10 into the video processing unit 12. The middle 
column labeled video out represents the sequence of outgo- 
ing frames after processing is done. The rightmost column 
indicates the total number of frames deleted. Tht process 
begins by specifying with manual control unit 19 the number 
of frames to be deleted or the time value of these frames to 
the A segment unit 18. Thereafter, the first four frames 
(F1-F4) are simply passed through the processing unit 12 
e.ssentially unaffected. Frame 5 (F5), however, is deleted and 
replaced with frame 6 (F6), and frames 7-9 (F7-F9) are 
output in sequence after frame 6, Similarly, frame 10 (FIO) 
is deleted, and frame 11 (Fll) is output after frame 9 (F9). 
After five frame times, one frame is deleted; after ten frame 
times, two frames are deleted, etc. up until the desired total 
number of fi:ames n (or the time corresponding thereto). 
Thereafter, the frames are simply passed through the digital 
video memory 15 essentially unaffected (since the total 
desired amount of time has already been accumulated). 

During the fi-ame deletion process, corresponding seg- 
ments of audio are similarly deleted. However, the audio 
segments need not correspond exactly to the frames deleted. 
Stated differently, portions of audio from one frame may be 
deleted along with portions of audio from a preceding or 
succeeding frame; or all of the audio of a given frame may 
be deleted, as desired. The manner in which the audio 
segments are actually chosen for deletion will depend upon 
the frequency characterLstics of the audio encountered, and 
are chosen in order to minimize the introduction of any 
audible noise signals into the final output signals, 

FIG. 3 is a schematic diagram illustrating the frame 
insertion process in order to expand the total run time of the 



45 



50 



60 



65 



03/01/2004, EAST Version: 1.4.1 



us 6,4; 

5 

program material. This process is essentially the reverse of 
the frame deletion process and proceeds by specifying the 
total number of frames or the time equivalent to be inserted 
into the length of the program material using manual control 
unit 19, followed by processing of the successive frames of 
video (and corresponding audio) to repeat every ith frame 
until the total number of frames (i.e., the desired time) have 
been accumulated. 

Appendix A to this application contains a discussion of 
the time equations which apply to proper operation of the 
invention. 

As noted above, A segment circuits 18, 28 can be manu- 
ally overridden by manual control unit 19 to provide opera- 
tor controlled frame insertion or frame deletion. In the 
preferred embodiment, manual control unit 19 includes a 
rotatablc knob with a detent feel. Rotation of the knob in the 
clockwise direction provides one inserted frame per detent; 
while rotation of the knob in the counter-clockwise direction 
results in one frame deletion. Not illustrated in the figure is 
a display unit, which may be any one of a number of 
conventional display devices (e.g. an LCD display) which 
indicates the total number of frames or total time value 
selected by the manual control unit 19 and, if desired, the 
running total of AV and AA. 

Returning to FIG. 1, a third mode of operation — termed 
auto mode — is also provided according to the invention. Id 
this mode of operation, the total number of frames or total 
amount of time to be deleted or inserted is again specified by 
manual control unit 19: however, the actual choice of which 
particular frames are to be deleted or inserted and which 
particular audio segments are to be deleted or inserted, is 
automatically controlled by a pair of detector circuits. Con- 
trol of the video frame deletion/insertion is done by a motion 
detector circuit 40 which incorporates any one of a number 
of known algorithms for determining the amount of motion 
change between adjacent frames, and permits deletion/ 
insertion of a given frame whenever the change in motion 
between the frames does not exceed a selected threshold 
value. Such circuits are well known and have been tised in 
video compression and coding systems. However, motion 
detect circuit 40 is constrained to either delete or insert a 
specified total number of frames over a fixed period of time 
in accordance with the parameters specified by manual 
control unit 19. Consequently, motion detect circuit 40 is 
provided with the accumulated total count from circuit 20, 
and an internal timing unit (not illustrated) in order to 
measure the progress of the frame deletion or insertion 
processing. If the total number of deleted or accumulated 
frames runs behind the elapsed real time (due to program 
content with relatively large amounts of motion over a large 
sequence of frames), the motion detect threshold is auto- 
matically raised by motion detect circuit 40 in order to 
permit a relatively larger number of frames to be deleted or 
inserted so that the system will succeed in deleting or 
inserting the desired amount of time over the prescribed total 
program real time period. 

Similarly, a pitch and level detect circuit 50 selects which 
audio portions contain the most effective frequencies and 
amphtudes capable of being deleted with minimal impair- 
ment to the audio (e.g. by not introducing "pops" or "clicks" 
into the audio). Pitch and level detect circuit 50 is similarly 
supplied with the running total from the AA circuit 30, and 
is provided with a threshold adjusting circuit to enable the 
threshold to be raised if the audio deletion processing is 
running behind the total elapsed time of the real time 
program. 

The sensitivity threshold of circuits 40 and 50 may also be 
functionally coupled to the amount of time change desired, 
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as suggested by the diagonal anows overlying elements 12, 
22, 40 and 50. Thus, for a maximum amount of time change, 
the sensitivity thresholds arc raised, while for a minimum 
amount of time change, the sensitivity thresholds are low- 
5 ered. 

Both motion detect circuit 40 and pitch and level detect 
circuit SO are provided with control output lines 42, 52 
which are used to control the A segment circuits 18, 28 on 
an on-the-fly basis. Thus, for example, motion detect circuit 

-jQ 40 may determine that three successive frames are to be 
deleted from the frame sequence: in such a case, a control 
signal is issued on control line 42 to the A segment circuit 
18 to delete the three identified successive frames. Similarly, 
pitch and level detect circuit 50 win determine those audio 

j5 segments which are to be deleted from frame portions, and 
issues control signals on control line 52 to the A segment 
circuit 28. 

In order to ensure that the total time delay between the 
time changed output video on terminal 17 and the time 

20 changed output audio on terminal 27 does not exceed the 
preselected maximum frame difference (i.e., ±1 frame time 
in the preferred embodiment), the accumulated video time 
and accumulated audio time are coupled from the a AV 
circuit 20 and AA circuit 30, respectively, via motion detect 

25 circuit 40 and pitch and level detect circuit 50 to an AA'' 
phase difference comparator 60. In the event that the video 
portions and the audio portions on the output terminals 17, 
27 are close to the maximum separation difference, the 
comparator 60 issues control signals on control lines 61, 62 

30 to the motion detect circuit 40 and pitch and level detect 
circuit 50. These control signals are then used by circuits 40. 
50 to select video frames and audio segments for deletion or 
insertion which steer the A/V difference in the proper 
direction. 

35 Some program materials may include segments which 
should not be subject to frame deletion or insertion due to 
the nature of the subject matter. For example, some com- 
mercial providers may require that the commercial program- 
ming information not be altered in any way. In such cases, 

40 it is useful to provide a pause function which terminates 
frame deletion and frame insertion until the sensitive pro- 
gramming material has been passed through the signal 
processing units 12, 22. This function is provided by dis- 
abling operation of the A segment circuits 18, 28, as well as 

45 operation of the manual control 19. The pause function may 
be implemented in the form of a manually operable switch — 
e.g. a push button switch — and an indicator specifying the 
state of the pause function circuit; or may be automatically 
implemented using appropriate circuitry for detecting pre- 

50 determined codes or other markers in predetermined por- 
tions of the video frames. For example, a special character 
may be inserted in the vertical blanking interval specifying 
the first frame in a sequence of frames for which the pause 
function is required; and an end of sequence special char- 

55 acter may be inserted in the last frame. Such a character may 
be conveniently inserted during non-viewable portions of a 
frame, such as in the vertical blanking interval. Other 
specific implementations of the pause function will occur to 
those skilled in the art. 

60 As will now be apparent, the invention is capable of 
increasing and decreasing passages of time in programming 
material by significant amounts, without impairing the sub- 
jective quality of the programming materials as viewed. For 
example, for NTSC video, by deleting one out of every 

65 twelve frames, a total of five minutes per hour can be 
accumulated or "saved" for other purposes. Similarly, by 
adding an additional frame every six frames in NTSC video 
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ten minutes per hour can be added to the total running time 
of programming material. In addition, it should be noted that 
the removal of audit segments at different points in time 
from the video frames optimizes the quality of the final 
video/audio output from the system, since it enables separate 
alteration of the video and audio portions based on the 
information content and using techniques which are optimal 
to video signals and audio signals separately. This ensures 
that the quality of the finally produced programming mate- 
rial is nearly as high as the original material. 

While the above provides a full and complete description 
of the preferred embodiments of the invention, various 
modifications, alternate constructions and equivalents may 
be employed, as desired. For example, while the invention 
has been described with reference to deletion and insertion 
of frames of video, these principles apply to deletion and 
insertion of individual fields of information. Therefore, the 
above description and illustrations should not be construed 
as limiting the scope of the invention, which is defined by 
the appended claims. 

APPENDIX A 

For this machine the following time equations must hold 
true: 



and 



AT^^ATv^tv 



10 



20 



25 



30 



Where: 

ATv=Total accumulated Video delay (+/-) time 
AT^=Total accumulated Audio delay (+/-) time 
Atv-Time duration of each Video packet (usually one 

frame) 35 
Atx=Time duration of each Audio packet (about 5 mil. sec. 

for this example) 
n- Number of Video packets needed to achieve total time 

change desired 
a=ratia between the time duration of the video and audio 

packets. 
For the machine to work: 



1) 



45 



2) 

These statements must hold to accomplish the desired 
time change (+/-) while maintaining lip-sync. Equation 1) 
holds for exact lip -sync while Equation 2) holds for mini- 
mum picture and sound anomalies. 
"Time Machine" run modes: 

Manual — ^whcre a knob can be turned clock wise or 
counter clock wise to increase or decrease time in 
frame increments. 
Auto — where a preset amount of time change (+/-) will 
automatically accrue where the audio and video can be 
locked together (ATv«AT^) or run independently 
(AT^oAT^At^) 
"Time Machine" features: 

Time change accrual can be stopped and restarted to 
coincide with such sacred timed segments as commer- 
cials. 

Motion, pitch and level change sensitivity can be adjusted 
to speed up or slow down the rate of time change 
accrual. 
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60 
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Maximum number of packet chances per second can be 
set to minimize excessive changes in time in the Auto 
mode. 

What is claimed is: 

1. An apparatus for altering the total running time of an 
original program signal in real time to generate a time- 
altered program signal, the original program signal includ- 
ing an original video program signal that includes a plurality 
of video signal elements in sequence, each video signal 
element having a predefined length of video time associated 
therewith, the original program signal also including an 
original audio program signal, the original audio program 
signal including a plurality of audio signal segments in 
sequence, each audio signal segment having the predefined 
length of audio time associated therewith, each video signal 
element having one or more corresponding audio signal 
segments associated therewith, the apparatus comprising: 

(a) a video analog-to -digital converter (ADC) that 
receives the original video program signal and converts 
each video signal element of the original video program 
signal to corresponding digital video data; 

(b) a video data processor that compares digital video data 
received from the video ADC and corresponding to 
adjacent video signal elements in said sequence of 
video signal elements, determines an amount of motion 
diEfcrcnce between said adjacent video signal elements, 
and deletes digital video data corresponding to a video 
signal element if the motion difference between said 
adjacent video signal elements is less than or equal to 
a predefined motion difference threshold value to pro- 
vide a time-shortened sequence of digital video data; 

(c) a digital video memory buffer that stores digital video 
data received by the digital video memory buffer from 
the video data processor; 

(d) an audio analog-to-digital converter (ADC) that 
receives the original audio program signal and converts 
each audio signal segment of the original audio pro- 
gram signal to corresponding digital audio data; 

(e) an audio data processor that compares digital audio 
data received from the audio ADC and corresponding 
to adjacent audio signal segments in said sequence of 
audio signal segments, determines an amount of audio 
signal difference between said adjacent audio signal 
segments, and, for each video signal element deleted 
from the original video program signal, deletes digital 
audio data corresponding to one or more audio signal 
segments, such that the amount of deleted audio time 
substantially corresponds to the predefined length of 
time associated with the deleted video signal clement, 
based upon the amount of audio signal difference 
between adjacent audio signal segments to provide a 
time-shortened sequence of digital audio data; 

(f) a digital audio memory buffer that stores digital audio 
data received by the digital audio memory buffer from 
the audio data processor; 

(g) a video digital-to-analog converter (DAC) that con- 
verts a sequence of digital video data output by the 
digital video memory buffer to a corresponding time- 
shortened analog video program signal; 

(h) an audio digital-to-analog converter (DAC) that con- 
verts a sequence of digital audio data output by the 
digital audio memory buffer to a corresponding time- 
shortened analog audio program signal; 

(i) circuitry that provides the time -shortened analog video 
program signal and the time-shortened analog audio 
program signal as a time-shortened broadcast program 
signal; and 
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(j) a threshold adjustment element for raising the motion 
difference threshold value on-the-fty such that, if the 
total number of deleted video signal elements runs 
behind a programmed elapsed time, whereby the 
motion difference threshold value is automatically 
raised to permit a larger number of video signal ele- 
ments to be deleted. 
2. An apparatus that automatically alters the total running 
time of an original program signal in real time to generate a 
time -shortened program signal, the original program signal 
including an original video program signal that includes a 
pluraUty of video frame signals in sequence, each video 
frame signal having a predefined frame time associated 
therewith, the original program signal also including an 
original audio program signal, the original audio program 
signal including a plurality of audio signal segments, each 
audio signal segment having the predefined audio segment 
time associated therewith, each video frame signal having 
one or more corresponding audio signal segments associated 
therewith, the apparatus comprising: 

(a) a video analog-to-digital converter (ADC) that 
receives the original video program signal and converts 
each video frame signal element of the original video 
program signal to corresponding digital video data; 

(b) a video data processor that compares digital video data 
received from the video ADC and corresponding to 
adjacent video frame signal in said sequence of video 
frame signals, determines an amount of motion differ- 
ence between said adjacent video frame signals, and- 
: automatically deletes digital video data corresponding 
to a video frame signal if the motion difference between 
said adjacent video frame signals is less than or equal 
to a predefined motion difference threshold value to 
provide a time-shortened sequence of digital video 
data; 

(c) a digital video memory buffer that stores digital video 
data received by the digital video memory buffer from 
the video data processor; 

(d) an audio an alog-to -digital converter (ADC) that 
receives the original audio program signal and converts 
each audio signal segment of the original audio pro- 
gram signal to corresponding digital audio data; 

(e) an audio data processor that compares digital audio 
data received from the audio ADC and corresponding 
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to adjacent audio signal segments in said sequence of 
audio signal segments, determines an amount of audio 
signal difference between said adjacent audio signal 
segments, and, for each video frame signal deleted 
from the original video program signal, automatically 
deletes digital audio data conesponding to one or more 
audio signal segments, such that the amount of deleted 
audio time substantially corresponds to the predefined 
length of time associated with the deleted video frame 
signal, based upon the amount of audio signal differ- 
ence between adjacent audio signal segments, to pro- 
vide a time-shortened sequence of digital audio data; 

(f) a digital audio memory buffer that stores digital audio 
data received by the digital audio memory buffer from 
the audio data processor; 

(g) a video digital-to-analog converter (DAC) that con- 
verts a sequence of digital video data output by the 
digital video memory buffer to a corresponding time- 
shortened analog video program signal; 

(h) an audio digilal-to-analog converter (DAC) that con- 
verts a sequence of digital audio data output by the 
digital audio memory buffer to a corresponding time- 
shortened analog audio program signal; and 

(i) circuitry that provides the time-shortened analog video 
program signal and the time-shortened analog audio 
program signal as a time-shortened broadcast program 
signal. 

3. An apparatus as in claim 2, and further comprising a 
broadcast system that broadcasts the time-shortened broad- 
cast program signal substantially simultaneously with gen- 
eration of the time-shortened broadcast program signal. 

4. An apparatus as in claim 2, and further comprising 
means for combining an additional program signal with the 
time-shortened broadcast program signal to generate a new 
program signal. 

5. An apparatus as in claim 4, and wherein the additional 
program signal comprises a signal corresponding to adver- 
tising information. 

6. An apparatus as in claim 2, and further comprising a 
counter that counts total cumulative time deleted from the 
original video program signal. 
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METHOD AND APPARATUS INCLUDING A stfll furlhCT object of the present inveation is to provide 

SYSTEM ARCHITECTURE FOR * novel, process architecture which allows the user to 

MULTIMEDIA COMMUNICATION control program and select fte approi^te media combina- 

tion either before during the communication session. 

«, 1 r"!^'trIh'£S" ^' ' SUMMARY OF THE INVEmON 

filed Dec. 15, 1994, now abandoned. 

FiFi n OF TNVlTKmON ^ BppaiBXus and mcdiod for multimedia communica- 

FUiUJ UN vmi lUiN ^^^^ including voice, audio, text, stiU image, motion video 

The present invention relates to a method and apparatus and animated graphics which permits communications 

for improving the efficiency of electronic communication between imiltimedia transmitters and receivers and which is 

and, in particular, to a method and apparatus which can con^atible with multiple standard or customized coding 

communicate with available electronic desk top equ^ment, algorithmic signals such as H.261, MPEG, JPEG, EDTV or 

sudi as personal computers, workstations, video cameras, HDTV whereby multiple incon^iatihlc video coding cquip- 

television, VCR's, CD players and telephones and receive, meat employing different video coding algorithms can now 

store, process and send multiple forms of media information, communicate with each oth^ and which includes a recon- 

sudti as sound, image, graphics, video and data, both digi- figurable memory for selectively adjusting the internal file 

tally and algorithmically based on a plurality of selective format and size so as to be c<Hnpatible with any available 

band widths. band widlh. 

BACKGROUND OF THE INVENTION 20 BRIEF DESCRIPTION OF THE DRAWINGS 

Technology aUows the individual to communicate widi ^^^^ present invention will 

others not only by the telephone, but also by telefax become apparent particularly when taken in view of the 

machines, personal computers and workstations utilizing f^uowing iUustrations wherein: 

modemsandtd^honelinesanddataandv^ FIG. 1 is a pictorial iUustration of the communicaaon 

can also be stcred and disseminated by means of videotapes, " * " 

compact discs and television monitors. system, 

•mere are methods and apparams available which aUow FIG. 2 ^ a schematic dmgram illustrating die overaU 

for large amounts of data to be reduced and transmitted in a ^V^*^ methodology; 

vciy short amount of time, such medxods and apparatus arc HG. 3 Is a schwnatic of the controUcr' s mtemal operatmg 

known as con^ircssing the data. Similariy, there are methods mode for illustrating band width management; 

and s^aratus available for enhancing the image quality of FIG. 4 is a schematic of the internal circuitry of the 

visual and graphic data that has been compressed and is now multimedia communications assembly; 

being displayed. For exaii^le, see U.S. Pat No. 4,772,947 pjQ 5 js a sdiemalic of the network communications 

to Kono; U.S. Pat No. 4,703350 to Hinman; U.S. Pat No. ^ proccssw and its design relationship to the transmission 

4,727,589 to Hirose; U.S. Pat No. 4,777,620 to Shimoni; processor; 

U.S. Pat No. 4,772,946 to Hammer; and U.S. Pat. No. ^ ^ schematic illustrating tfie communication 

4398256 to Nussmiei. between tfic host processor, system memory, pixel jMrocessor, 

While the afracsaid patents teach various methods and ^^^^ memOTy and display processor; 
apparatus for compressing and decompressing data and ^ piG. 7 is a schematic of die video codec and display 
enhancing the image quality of the data, none of the aiorc^ 

said patents have directed themselves to the concept and JJ!! ^ ^ .„ ♦ ^ ^ r-ro «nH 

stmcJuTe of a method and aM>aratus which would commu- FIG. 8 is a sdicmatic lUustration of the standard OF and 

nicate with and share resources among the telephone, per- QP^ memory format; 

sonal computer or workstation, video screen and VCR to 45 HG. 9 is a schematic iUustration of applicant's scalable 

allow the individual to select and convey multiple forms of mcmcny array reconfigurable technique; 

media information such as sound, image, graphics, data and fiq, 10 is a schematic illustrating the pixel processor 

live video In an efficient and effective architecture which flexibility to various video coding algorithms; 

would automatically adjust to available band widths and ^ schematic of the motion processor sub- 

whidi would be capable of communicating in nuiltiple band systems: 

FIG. 12A illustrates a programmable logic device 
OBJECTS OF THE INVENTION eno^loying cellular array logic architecture: 

An object of the present invention is to define an ime- HG. 12B illustrates the implemenUtion of ceUular logic 
grated process architecture which can accommodate 55 proccssmg; 
communications, both transmission and retrieval, of all FIG. 12C illustrates gating logic; 
digitally-coded or algorithmic multimedia infonuation. piG. 13 is a sdiematic of die multimedia assembly. 

Another object of the invention is to P^^^^/^ » »«^^ DETAILED DESOUPnON OF TEE DRAWINGS 

system architecture which is flexible and allows control of i/c4,rk*i-ca>F m^^^k.^ 

me variable communications band wid&s and allows for 60 Ref cuing to FIG. 1, there is shown a pictorial illustration 

flexible combinations of digitally^coded multiple media depicting the oonununication devices available presentiy for 

information having application to teleconferencing or cdu- the home or office. These include a VCR 102, CD player 

cational instruction. 103, telephone 104, television 106, personal computer 108 

A still further object of the present invention is to iffovidc and fax machine 110. Each of these communication devices 

for a novel process architecture which not only allows for 65 has a distinct function. The telephone can transmit and 

digital coding techniques, but also can interface with Iradi- receive audio and data; a fax machine can transmit and 

tional analog storage or transmission techniques, receive text documents, a television can receive video broad- 
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casts and audio; and a personal computer can be used to 
many data processing ^plications. It is Applicant's inten- 
tion to disclose an assembly which can physically commu- 
nicate with these electronic devices to permit them to 
function complimentaiy with each other and to communi- 
cate wiHi other electronic devices regardless of whether the 
other communication devices were digital or algorithmic 
and to code and decode autraaatically to the available band 
wid&. The communication is acconq>lished by a multimedia 
communications assembly 112, being of size and shape, 
similar to that of a VCR. The aforementioned electronic 
devices would int^connect with the multimedia communi- 
cations assctubly 112 to allow the user/operator to control, 
con^lement and utilize the functions of the electronic 
devices by means of the multimedia communications assem- 
bly 112. 

FIG. 2 Illustrates the overall system operation and meth- 
odology for ^e multimedia communications assembly 112. 
Assembly 112 makes it possible to exchange a multitude of 
different forms of media objects over a wide range of 
conomunication networks. Mor art has shown methods and 
^aratus to imptovc compression and decompression tech- 
niques for individual media types and individual band width 
ranges. However, since video coding algosithms are intrin- 
sically incompatible with each oth^, there is need for an 
asseznbly 112 to provide a common interface whereby 
incojiq>atible equipment can freely exchange media objects 
throu^ interfacing with assembly 112. 

The sdiematic methodology illustrated in HG. 2 com- 
prises the following major system continents. They are a 
network conozuinications processor 202; a transmission pro- 
cessor 204; a pixel processes 206; a motion jHOcesscn: 208: 
a transform processor 210; a display processor 212; a 
capture processor 220; a &ame memory 214 and a host 
processor 218. 

The design of the system architecture as described in 
detail hereafter is to gain the ability to interface with 
multiple types of media objects, inchiding audio, still image, 
motion video, text and graphics. As illustrated in FIG. 2, 
gr^Thics input might possibly be in the form of an RGB 
format 224; VGA format 226; XGA format 228; or SVGA 
format 230. Text media objects could be either in the form 
of a Group 3 format 232; Groii^ 4 format 234; or ASCI 
format 236. Motion media objects may conform either to 
H.261 fonnat 238; MPEG format 240; or other specialized 
formats 242. Still background media objects could be con- 
forming either to JPEG format 244 or other specialized 
formats 234. Input audio media objects could be confoim- 
fing to CD audio format 246; voice grade audio 248 or FM 
audio format 250. 

Each media object wilhin a category, namely, audio, still 
image, motion video, text and graphics would be imported 
to a multiplexer 252 dedicated to each category in wder to 
identify the input signal and then be directed to a dedicated 
overlay 254 for each category of media object The overlay 
254 provides the ability f<x the assembly, disassembly, 
deletion, addition and modification of a selected group of 
miiirimwHa objects. The input signals, be they audio, stili 
image, motion video, text or gr^hics, are converted into 
conq^uter object-oriented language format for encoding into 
a frame menK>ry 214 as described hereafter. This conversion 
before storing into frame memory 214 In cooperation witti 
the major components of the system described hereafter, 
permit the compilation of selected ii^ut signals which have 
been stored In the frame memory 214 to be assembled, 
interpreted and translated to other system formats wilh 
relative ease as a result of the intelligent memory manage- 
ment capability inherent in this design. 
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The system architecture provides for an interface which 
will enable nuiltiple inco^^3at^ble video coding equipment 
eiiq>loying different video coding algorithms to communi- 
cate. This is accon^lished through a scalable frame memory 
5 architecture reconfigurable technique (SMART) described 
in FIG. 9. 

In simplistic tenns to be described in detail hereafter, the 
design of assembly 112 allows host processor 218 to identiiy 
the types of input articles during the import stage, the host 

10 processor will then instruct the reconfiguration circuit 256 
and the scaler ctrouit 258 to provide the required down- 
sanq>ling ratio. The media article being in^Kirted can then 
conform or be reduced to the internal file format during the 
imp<^ stage. The reverse is true during the exporting stage 

^2 when the media artide in the internal file can be enlarged 
and made to conform to the apprc^niate algorithm for the 
exporting stage. As a result of our smaller internal file size, 
the real time performance requirement of our pixel processor 
206, graphics processor 222, transform processor 210 and 

20 motion processor 208 is reduced. Further, the speed and size 
of the frame memory 214 is also proportionately reduced. 
This design allows various coding algorithms to be micro- 
coded at pbid ]»xxxssor 206. 

Assembly 112 also optimizes die video coding for specific 

^ coixq>ression ratios in order to meet specific band width 
rcqufrcments. In order to adjust the band width to meet the 
various coimrmnication network requirements, band width 
controller 260 receives the band widdi requirement from the 
netwctfk communication processor 202, the band width 

^ controller 260 will then instruct the host processor 218 to 
develop the appropriate coo^iression ratio in order to meet 
the real time performance rcquircxneots. Band width con- 
troller 260 will also interface with transmission processor 
204 in order to import and export the media artide at die 
appropriate band width. Assembly 112 can program die 
network communication processor 202, die transmission 
processor 204 and the display processor 212 to provide die 
various types of communicadons interfaces. 
The internal operation modes of host processor 218 

^ permit it to adapt to different conqaression ratio requirements 
and network band width requirements. As an example, the 
following are some popular n^work band widdi interfaces: 

1. Communicating over an analog phone line en^loying 
V32 modem, 9,600 bit per second (bps) band width is 
required, a quarter common immediate frame (QCIF) format 
is displayed at 7.5 frames per second (^s). 

2. Communicating over a digital ISDN D channel at 16 
Ulo bits per second (kbs). The user has two <^tions, dthcr 
two quarter comiiK>n intermediate frame (QOF) formats can 
be displayed at 7.5 frames per second or one quarter 
conunon intermediate frame can be displayed at 15 frames 
per second. 

3. Communicating over an analog phone line whereby 
33 19,200 bit per second band width is required The user has 

two options, either two QOF (common intermediate frame) 
formats can be displayed at 7.5 fi^mies per second or one 
QOF (quarter common intermediate frame) can be dis- 
played at 15 fr^es per second. 
60 4. Communicating over switched 56 Idlo bits per second 
digital network. (Quarter common intermediate frames with 
three quality level qitions will be updated at 15 frames per 
second. 

5. Communicating over a single ISDN B channel over an 
65 ISDN basic rate interface network, four quarter common 
intermediate frames will be concurrently updated at 15 
frames per second. 
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6. Communicatiiig over a dual ISDN B channel in a ISDN 
basic rate interface network, quarter common intermediate 
frames will be transmitted at 30 frames per second. 

7. Communicating over a 3S4 Idlo bits per second ISDN 
HI network, common intermediate frame Will be transmit- 
ted at 15 frames per second. 

8. Conununicating over a 1.544 kflo bits per second Tl 
network, common intermediate frames (QF) will be trans- 
mitted at 30 frames per second 

As a result of the aforesaid plurali^ of band widths, It is 
necessary for the multinLedia assembly to continuously 
monitor the processor and network band widdi availability 
and to simultaneously detennine the amount ol con^sression 
or decompression that is required with respect to the data in 
frame memory 314 to be transmitted. Due to the variable 
band width or throughput requirement for each transmission 
network, only dedicated processor ^rproaches have been 
shown in the prior art to meet a specific band width 
pcrfotmancc. For example, three video conferencing tech- 
niques are required at the 112 Kbs, 3S4 Kbs and 1.544 Mbs 
band width range. The multimedia assembly disclosed 
herein, includes different transceiver pairs for each specific 
netwode type. The system architecture disclosed herein, and 
in particular, host processor 218 in conjunction with band 
width controller 260 unit, scaler 258 and reconfiguration 
unit 2S6, can continuously adapt to a variety of network and 
processor band width c^ianging situations, for exan^e, 
noisy local line condition and network traffic congestion. 
This is possible as a result of the scalable memory archi- 
tecture which permits the continuous reprogramming of the 
internal file format of frame memory 214 so tbat it is suitable 
for the specific band widdi requirement at that moment 

During the interframe coding mode 278, after the incom- 
ing media articles are received, the ^ipropriate firame size 
262 will be adjusted first, frame by frame difference 264 will 
then be calculated. For consecutive frame processing, an 
appropriate motion vector 270 can be derived. For selective 
frame processing, due to the difficulty to identify a suitable 
motion vector 270, interpolation techniques 266 can be 
en^iloyed to simulate frame difference signal. Decision 
logic 272 is employed to analyze situation and make a final 
decision. In case of scene changes, system will be reset to 
intraframe coding mode for iurther processing. A detailed 
design the motion processor 208 is frulher shown in FIG. 
11. 

Although our invention entitled '^multimedia'*, we have 
been mostly focued on "new hardware and software tech- 
niques" for the **nx)tion video**, In addition, we have also 
shown new techniques how to int^rate (overlay) motion 
video with other media article in order to create a complete 
multimedia presentation. Since there have been plenty of 
prior arts showing techniques to handle other media, ie., CD 
audio, fax, telefdione, computer gr^>hics, or digital camera. 
Also because tiie performance requirement for these media 
types are much less demanding. Therefore, the encoding and 
decoding of other media types in our invention can be easily 
implemented in general purpose computer hardware and 
software, embedded hardware controller, or special purpose 
digital-signal processors. 

FIG. 3 is a schematic illustration of the controller's 
operating modes for band width management based upon the 
intexnationai compression standard CCITT H.261. Based 
upon this standard, each conmion intermediate format frame 
(GIF frame) 302 consists of twelve (12) groups of blocks and 
each group of blocks would consist of thirty-three (33) 
macro-Hocks with eadi macro-block consisting of six (6) 
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blocks (4 Y's and2 UA^*s), Each block would consist of 8x8 
pixels and each pixel would consist of an 8 bit value. The 
quarter conomon intermediate format frame (QCIF frame) 
304 would consist of three groups of blocks and these would 

s be identical to those of the CIF*s 302. 

In multimedia assembly 112, host processor 218 has eight 
(8) different network interface modes 306. The first interface 
mode 310 is for 9.6 Kbs analog modems. The second 
interface mode 312 is for 16 Kbs ISDN D channel. The third 

to network interface mode 314 is for 19.2 Kbs high speed 
analog modems. The fourth network interface mode 316 is 
for 56 Kbs digital network (PSDN). The fifth network 
interface mode 318 is for 64 Kbs ISDN single B channel. 
The sixth network interface mode 320 is for dual B channel 

15 128 Kbs ISDN BRI nrtwork. The seventh network interface 
mode 322 is for 3S4 Kbs ISDN HI network and the eighth 
network interface mode 324 is for 1.544 Mbs ISDN PRI or 
Tl nctWOTk, 

Host processor 218 also has progranmiable frame updat- 

^ ing rate c^ability 326. Frame updating rate 326 provides 
host processor 218 with five options. They can be either 30 
frame per second (fps); 15 ijps; 10 fjps; 7.5 fjps or 1 fps. 

The standard frame update rate 326 for each network 
interface mode 306 would be I ^s for first network interface 

^ mode 310; 1.5 fps for second network mtcrface mode 312; 
2 ^ f<»: fiiird netwode interface mode 314; 6.5 fps for fourth 
network interface mode 316; 7.5 fps for fifth interface mode 
318; 15 fps for sixth and seventh interface mode 320 and 

^ 322, respectively and 30 fps for eighth interface mode 324. 
In FIG. 3, we have established 30 fps frame update rate 
326 as the default update rate for C!IF format 302 transmis- 
sion and 7.5 fps as the default update rate for QCTF format 
304 transmission. The con^iression ratios illustrated in FIG. 

^ j 10 and described hereafter are for this default update rate. 
The CIF format 302 system througtquit requires 4.6 mega 
bytes per second (MBS). The QCIF formal 304 requires 2S8 
kilo bytes per second. Assuming we use 8 kilo bytes per 
second as the measuring base for real time video transmis- 

40 sion over iiJtti network interface mode 318, the CIF format 
302 system would require a ccmquression ratio of 576:1 
based upon the CCTIT H.261 compression standard. The 
QCIF format 304 would require a 36:1 compression ratio. 
Similarly, with respect to the other network interface modes 

45 306, the compression ratios would be as follows: The eighth 
network interface mode 324 would require a CIF format 302 
conqiression ratio of 24:1 whereas QCIF format 304 would 
require a 1.5:1 compression ratio; seventh network interface 
mode 322 would require a CIF format 302 compression ratio 

50 of 96:1 and a (JCJF format 304 ratio of 6:1; fourth network 
interface mode 316 would require a CIF fcnmat 302 com- 
pression ratio of 658:1 and a (^CIF format 304 ratio of 41:1; 
third network interface mode 314 would require a QF 
format 302 con^ression ratio of 1,920; 1 and a QCfP format 

55 304 ratio of 120:1; the first network interface mode 310 
would require a GIF format 302 ratioof 3,840:1 andaCJCDP 
format 304 ratio of 240:1. 

As a standard operation in Applicant's multimedia 
assembly, single QCIF format 304 will be en^loyed for ttie 

60 first through fifth network interface modes 310, 312, 314, 
316 and 318, re^ectively. Double OOF format will be 
employed for sixth network interface mode 320 and single 
OF format 302 or quadruple QCIF format 304 sequences 
will be utilized for the seventh and eighth network interface 

65 modes 322 and 324. 

The advantages of Applicant's multirxKdia communica- 
tions assembly 112 and its operation and capabilities will be 
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discussed hereafter. FIG. 4 illustrates a schematic view erf YUV 471 4:2:2 472. The ability of the capture iMx>cessor 220 

the multimedia communications assembly 112. It consists of to decode the afoiesaid formats provide for a convenient 

the following major system components. They arc a network interface between the mnlHmf d ia communications assembly 

communications processor 202; a transmission processor 112 and the television 106, VCR 102 or video camm 

204; a pixel processor 206; a motion processor 208; a j The GIF 302 formulated YUV 471 signals will first 

transf<Hin processor 210; a display processcr 212; a capture transfer out of the capture processor 220 and store in the 

processor 220; a frame memory 214 and a host processor firaine memory 214. The luminance (Y) signal 474 will be 

218. These system components can be inqdemcntcd ctther loaded into ^e motion processor 208 to perform motion 

using custom integrated circuit devices, a programmable estimation 475. A motion vector 476 will be developed for 

integrated circuit; microprocessor, midocontroUen digital each maao blodc 477 and store in the associated frame 

signal processor or software, d^nding upon the specific memory 214 location. The dififecence between the new and 

system perfonnance requirement old macro blocks will also be coded in discrete cosine 

The system components arc interconnected through a transfonn (DCT) coefficients 478 using the transform pro- 
system host bus 418 and a high speed video tws 422. The cesser 210. Pixel processor 206 will perform a raster to 
system host bus 418 allows the host processor 218 to control zig-zag scan conversion 460, quantization 462 and VLC 
access and communicate with the system components such coding 458 of the DCT coefficients 478 for each macro 
as the n^crk communication processor 202, the transmis- block 477 of luminance 474 and chrominance 473. The 
sion processor 204, die pixel processor 206, and the frame transmission jooccssw 204 will format the QF 302 frames 
memory 214. The video bus 422 interconnects die frame into the CCUT H.261 238 format and attach the appropriate 
memory 214 with such con^nents as the cqjture processor ^ header 481 infoimaticm. As an cxan^lc, a CIF frame 302 
220, the display processor 212, the transftam processor 210, will partition into twelve groups of blocks 482 and each 
the pixel processor 206 and die naotion processor 208 to group ofblocks 482 will consist <rf thirty-three macro blocks 
perform high speed video signal processing functions. Bodi 477 and each macro block 477 will be conqwsed of four 
the system host bus 418 and the video bus 422 arc luminance signals 474, and one U & V signal 473. The 
bi-directional parallel buses. 25 i^^^^ communication processor 202 will i^ovide the 

Due to the real time prfOTmance requirements for the control interface to die telecommunications network 480 or 

hi^ ^ed video frame i»occs&ing, two system-wide inter- to a miCTOwave link 483. 

connections are In^lcmented. The first is die video pq>eline On the receiving side, the serial con^sscd video bit 

424 consisting of a direct interconnection between the stream 484 will be received from the network oommunica- 

capture processor 220, pixel processor 206, motion proccs- 30 tion processor 202. The bit stream will be converted from 

sor 208, transform processor 210, frame memory 214 and serial to parallel and decode the apjMopriate header message 

display processor 212. The second system interconnect 342 481 using the transmission processor 204. The infarmalion 

consists of the direct interconnection between the network will then be scot to die frame memory 214 dirough pixel 

cormnunication processor 202. transmission i^ocessor 204, processor 206. Pixel processor 206 will dien perform a 

host processor 218 and pixel processor 206. In otdex to 35 variable lengdi decoder 458, zig-zag-to-rast^ scan conver- 

fadlitale these interconnect operations, first in, first out sion 460 and dequantization 463. The YUV 471 macro block 

manory devices 428 are inserted where qjpropriate. 477 of DCT coefficients 478 will be sent to frame memory 

The frame memory 214 can be inqjlemenled either in 214 dirough laxel processor 206. Pixel processor 206 will 

static random access memory 430 or video random access then send YUV 471 macro blocks 477, one at a-time to the 
mraiory 434. The static random access memory 430 is easier 40 transfonn processor 210 to perform inverse DCT operation 

to implement, but at a higher cost The video random access 485. The YUV 471 diflFcrencc 450 will dien be added to die 

memory (VRAM) 434 is less expensive, but slower than die old signal 452 to conform to a new YUV pixel 446 for each 

static random access memory 430 and requires a controller macro blodc 477. The display jffoccssor 212 wiU then 

434 to update die memory array. The video random access perform YUV 471 to RGB 224 conversion and generate 
memory 434 is provided wi* two access pOTts 436 and 437 43 analog signal from &e RGB 224 or tixcacc generate an 8 bit 

providing access to die random accessible memory array. VGA 226 color image ttirougji color moping 486. The 

This is done since many video coding algoriduns employ di^lay processor 2U wiU Uien provide a convenient intcr- 

frequent use of die intetframe coding 440 to reduce band face to various displays sudi as television 106, personal 

widdis. Namely, only die frame difference signal 442 will be computers 108 or monitor. 

transmitted. Therefore, the twin memory accesses are 50 For ease of intofacc, host processor 218 also provides for 
required to store bodi the new frame 444 and the old frame a high speed smiall computer system interface (SCSI) 488 
448 and to facilitate frame differencing operations 450. In widi die external host 487 such as a personal con^>uter or 
tius design, die pixel processor 206 serves as the bus master work station. The advantage of the small computer system 
420 for die video bus 422 by having the video random access interface 488 is diat it provides a system independent 
memory (VRAM) controller 434 function positioned widiin 55 interface between die external host 487 and the multimedia 
die pixel processor 206 core. This allows pixel processor communications assembly 112. Since only simplified con- 
206 die ability to control video bus 422 and to access video trol messages 489 are required to pass between the two 
random access memory pixel stOTage for pixel level opera- hosts, modifications to the system to i^ovide for various 
tions 454. Pixel processor 206 also is equ4q)cd with the bit operation formats sudi as DOS 491, UNIX 490 or Macin- 
level manipulation functions 456 such as variable lengdi 60 tosh 492 can easily be accomplished. The higji speed small 
coder and decoder (VLC) 458, scan format converter 460 computer system interface 488 will also allow die transrais- 
and quantization converter 462. These permit the pixel sion of video sequences between die two hosts, 
processor to utilize international video coding algorithms for In die case of high speed digital network communication, 
communicating as discussed hereafter. the communication p^)eline is employed to facilitate real 
The capture processor 220 can decode various types of 65 time frame fcmnatting 410, protocol controlling 412, trans- 
analog video input formats and convert tiicm (e.g., NTSC mission and decoding. The host processor 218 is the bus 
464. PAL 466, SCAM 468, or SVHS 469) to CCIR 601 470 master 420 for die system bus 418. Consequentiy, host 
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processcH' 218 will be able to access to the £raine memoiy 
214 a&d/oi system menuay 216, and monitor progress 
through a windowing operation 4^. The windowing opera- 
tion 494 essentially allows a portion of the system memoiy 
216 to be memory mapped 49S to the frame memory 214 so 
Qiat the host processor 218 can use it as a window to view 
&ame memory 214 status and operations in real time. 

FIG. 5 illustrates the network communication processor 
202 and its design relationship to transmission processor 
204. Network communicatioa processor 202 is comprised 
an analog front end transceiver 514, digital signal processor 
modem 516 and a buffer memory 518. These network 
communicatioD processor 202 components are intercon- 
nected through a private NCP bus 520. The transmission 
processor 204 consists of a frame formatter 522, a protocol 
controller 524 and an error processor 526. The transmission 
processor 204 cono^nents and the buffer memory 518 are 
interconnected through another private X bus 528. The 
bit-serial D bus 530 facilitates the network communication 
processor 202 and transnoission jHOcessor 204 conmmnica- 
tion through digital signal processor modem 516 and frame 
fOTiatter 522 sub-systems. The private NCP bus 520, D bus 
530 and X bus 528 are designed to facilitate effective data 
addressing and transfer in between the sub-system blocks. 
Ftirthermore, the buffer memory 518« digital signal proces- 
sor modem 516 and protocol controller 524 are intercon- 
nected to the host processes 218 ttirougb system bus 418. 

The specific requirement of the bus design, which may 
include address 510, data 512 and control 502 sections is 
dependent upon the data tfarou^iput 504, word size 506 and 
bus contention 508 considerations. The network communi- 
cations pa^ocessor 202 implements the DTE 536 function 
while the host processor 218, and transmission processor 
204, perform the DCE 532 function. This allows the proper 
pairing <^ the DCE 532 and DTE 536 intofaced to a local 
customer premises equipment 534 so as to perform confer- 
ence control 538, store and forward 540 or band width 
management 542. 

Within the network communication processor 202 sub- 
system, digital signal processor modem 516 is &e local host 
controller 544. Analog front end 514 consists of an analog 
to digital converter (ADC) 546 and a digital to analog 
converter (DAQ 548. The analog-to^ligital converter 546 
samples and hcdds &e analog input signal 550 and converts 
it to a digital bit stream. The digital-to-analog converter 548 
buffers the digital ou^ut bit streams and converts them into 
an analog output signal The analog frtint end is the frx^nt end 
interface to the telephone network 480 from ttie system. The 
output digital bit stream from the analog-to-digital oonvoter 
546 is then transferred to the buffer memory 518 for tem- 
porary storage. The digital signal processor modem 516 will 
access this information through buffer memory 518 to 
perform line coding functions. Inside the digital signal 
processor modem 516 is a prograimnable digital signal 
processor 552. Digital signal processor 552 Is programmable 
allowing for easy implementation of line coding 554 and 
control 556 functions for many of the analog front end 514 
functions. 

Within the dransmission processor 204 sub-system, the 
frame formatter 522 first received die incoming information 
frame 558 and header message 481 from the digital signal 
processor modem 516 and identifies the proper receiving 
video coding algorifiim types 560. Protocol contr(^er 524 
then takes over and starts the appropriate protocol decoding 
562 procedures. Once &e control frame 564 and information 
frame 558 header information are fully decoded, the infor- 
mation frame 558 is sent to the error processor for error 
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checking and correction. Coirected bit streams are then 
converted from serial to parallel fana using serial to parallel 
converter 568 and are stored in the first in and first out buffer 
428 for further processing. The first in, first out buffer 428 

5 is designed into four 32K bits section. Each section allows 
fofC storage of 32K bits which is the maximum allowance of 
a conqnessed OF frame. Therefore, I28K bits in the first in« 
first out buffer allows double buffering and simultaneous 
transmitting and receiving of the incoming and outgoing 

iQ video information frames. 

In order to accommodate the various network 
enviroiunents, the network communications processor is 
designed to operate in the following specific speeds. 
9.6 Kbps (Kilo bits per second), 19.2 Kbps, 56 Kbps, 64 

15 Kbps, 128 Kbps, 384 Kbps, 1,544 Mbkps (mega bits 
per second) and 2.048 Mbps. HP will offer three 
options as the standard modes of operation. In mode 2, 
single GIF or four QdF sequences will be offered at 
384 Kbps and higher. In mode 3, two QCIF sequences 

20 will be offered simultaneously at 128 Kbps. 

When line conditions degrade, the analog front end 514 
will become aware of the degradation as a result of iocoming 
frame synchronous signal 570. Analog front end 514 will 
then notify the digital signal processor modem 516 and host 

25 processor 218. Host processor 218 wtU then switch from a 
standard operation to an exception operation mode. Host 
processor 218 has three (^ons to lower the bit rate in order 
to accommodate and correct the degradation. Option 1 
would be for die host processor 218 to notify die pixel 

30 processor 206 and select a coarser quantization level 57Z 
Option 2 would be to drop the frame update rale and increase 
the interpc^on rate 574. Option 3 would be to drop from 
OF to QCSF 576. When the error processor 526 detects 
more than two single bit errors, the error processor 526 will 

35 notify the pixel processor 206 and host processor 218. Host 
processor 218 again has two options. Hither pixel jvocessor 
266 can request for an retransmission or host processor 218 
can delete the con^lete macro block 477 and wait until the 
next macro block is sent. Meanwhile host processor 218 will 

40 send die old macro block 308 firom the fr-ame memory 214 
and use it to update the display. 

FIG. 6 illustrates the interactions between the front end 
communication systems and the host processor 218, system 
memory 216, pixel processor 206, frame memory 214 and 

45 display processor 212. These interactions are performed 
throu^ system bus 418. The incoming video sequence 602 
is first received by a front end demodulator 515. Network 
communications processor 202 and transmission processor 
204 will decode the condrol message and header information 

50 606. The pixel processor 206 and transform processor 210 
will then transform these sequences from frequency domain 
to pixel domain and store same in the frame memory 214. 
The display processor 212 perfonns the ^)[7opriate into-- 
polation to di^lay the output video sequence at the selected 

55 frame rate. Similarly^ the outgoing video sequence 6fl3 can 
be prepared dirough coding of die fr-ame difference 442 for 
each macro block 477 to convert from pixel to frequency 
domain to transmit out through frx>nt end modulators 514. 
Onoe the incoming video sequence 602 is received and 

60 stored in die buffer memory 518 the control message and 
header 606 information will then be stored in a first in, first 
out memory 428 for further decoding by the network oom- 
munications processor 202 and transmission processor 204. 
A self-contaioed miao controller 608 could provide the 

65 frame formatting 610. error processing 612 and protocol 
control functions 524. This would provide service at low bit 
rate applications iq> to 64 Kbs range. For higher speed 
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plications 16 tat or 32 bit high perfarmancc embedded before final addition. The double buffered iiq>ut 707 and 

micro cootroUexs could be employed. output 710 first in» first out nacmories and the iiqjut multi- 

FIG. 7 illustrates a block diagram of die design of the plcxa 716 are cn:q)loyed to allow the four stage pipeline 

video codec and display subsystem 702 and its interaction required far the discrete cosine transform operation. Addi- 

with the transmission processor 204 and host processor 21S. 5 tional speed may be obtained dmni^ the use of additional 

The video codec and display subsystem 702 consists of pixel transf onn pipeline processor 744 arranged in parallel, 

processor 206. transf onn processor 210, frame memory 214 Ref ciring to FIG. 8, as background to Applicant*s scalable 

and display proccssw 21Z Pixel process^ 206 is the host mem<Hy array reconfiguiat^e tedinique to be described 

controller for the video codec and di^lay sub-system 702. hereafter, an understanding of the OF format 302 and QCIF 

Pixel processor 206 is also the controller for the video bus lo fonnat 304 is necessary. These fwinats arc designed for die 

422. Pixel processor 206 cOTimunicates with the host pro- transportation of video information over a tcleoommunica- 

cessor 218 through system bus 418 using its internal host tion network. They are commonly applied by international 

interface circuit 704. Pixel processor 206 also intCTOonnects coding algorithms such as CCTTT H.261 238 and MMG 

to transmission processcK 204 dirough a first ia, first out 240 standards. 

memory buffer 706 using its internal serial interface 708. 15 The GIF format 302 consists of 352 jaxcls for each 

Pixel processor 206 interfaces and controls frame memoiy horizontal scan line with 288 scan lines on the vertical 

214 through video bus 422 using ite internal VRAM con- dimension. The OF fonnat 502 is further partitioned into 

troUer circuit 434. Pixel processor 206 interfaces with twelve groups of blocks 482. Each grotq) of block consists 

motion processor 208 dirougfi video bus 422 and with of 33 macro blocks 477 and each macro block consists of 4 

display processor 212 through private DP bus using its 20 Y blocks 474, 1 U block 473 and 1 V Wock 473 and each 

internal display processor decoder 714. The pixel processor block consists of 64 8-bit pixels. 

206 also interfaces with transform i«t)cessor 210 through The QCIF format 304 consists of 176 pixels for each 

first in, fii^t out memory 707 and vapxit multiplexer 716. horizontal scan line with 144 scan lines on the vertical 

Pixel processor 206 is also required to perfcam tinM dimension. The QCIF format 304 is further partitioned into 

critical pixel domain video coder and decoder functions 718. 25 three groups of blocks 482, each group of blodc 410 

These include variable length coder and decoder, run level consisting of 33 macro blocks 477 with each macro block 

coder and decoder, quantization and dequantization, zig-zag consisting of 4 Y blocks 474, 1 U block 473 and 1 V block 

to raster or rastar to zig-zag scan conversion. 473. 

Since most video coding algoriduns employ frame dif- Each macro block 477 comprises 384 bytes of YUV data 

ferencing 450 techniques to reduce band width, only the 3o since the frame rate for QF fonnat 302 is 30 fys (frame per 

frame diffffence signals 442 will require to be coded and second) and each OF fonnat 302 frame consists of 396 

decoded Frame memory 214 is designed to sU»e old frames macro blocks. The band width required to send uncom- 

714 and new frames 712 at two discrete section. Old frame pressed OF format 149 frames would be 4.6 mega bytes per 

714 being stored as the reference model while the difference second which is the equivalent to a total of 576 channels of 

between the new and old frames are being updated via a 35 64 Kbs B channels. 

differencing signal 442 which will be either coded for Each QCIF fonnat 304 has 99 maao blodcs 477 and 

transmission or decoded and added back to the old frame frame updates at 7.5 ^s. The system fliroughput requires 

714 for the reconstruction of new frame 309. 288 KBs which is the equivalent of 36 channels of 64 KBs 

As an encoder, pixel processor 206 will retrieve from the based B channels 80Z Therefore, an uncon^ssed OF 

frame memciy 214 these frame differencing signals 442 in 40 format 302 frame transmitting at 30 fps requires 24 Tl lease 

macro blodcs 477. Ttansfonn processor 210 will perform Ae lines 804 and the QCIF format 304 transmitting at 7.5 ^s 

DCr (disactc cosine transform) function 716 to translate requires 1,5 Tl lines 804. As such, 75 miao seconds would 

each of the Y, U, and V block from pixel to frequency be required to code an incoming GIF format 304, 1.2 

domain. The pixel processor 206 will ^ly these discrete milliseconds would be required for each macro block at 7.5 

cosine transforms to the decoder or encoder function t^eforc 4S fps. 

fcHwarding the coded bit stream to die transmission proces- The CCTTT H.261 standard 238 requires a switch from 

sor 204 for transmission. inter to intra frame mode aftCT every 132 frames of trans- 

As a decoder, pix^ processor 206 will retrieve these mission in order to avoid accumulative error. This means 

frame difference bit streams 442 from the transmission that in a 30 ^s transmission, every 4.4 seconds intra QF 

processor 204 first in, first out buffer 706, apply the decoding so format 302 frame coding will be engaged and in QCIF 

procedures, and then cormnunicate with the transform pro- format 304, at 7.5 ft>s, intra frame coding will be engaged 

ccssor 210 through its input first in, first out buffer 707. every 17.6 seconds. 

Transform processor 210 will pcifcnn the inverse DCT FIG. 9 is a schematic illustration of the scalable memory 
(discrete cosine transform) opaation 485 to derive the pixel array reconfigurable technique utilized by Applicant's nnil- 
domain values for each Y, U and V block 471. These pixel 55 timedia assembly 112 in order to optimize the performance 
values will be stored in the transform processor output first for encoding OF formal 302. To achieve 30 ^s updates, die 
ia» first out 710 until the pixel processor 206 retrieves the old time required to encode a macro block 404 is 75 microsec- 
pixcl block from frame memory 214. The signal differential onds. A single 8x8 DCT operation will consume 6.4 micro- 
will then be forwarded to the pixel processor to update the seconds. Since it takes 6 DCT operations to complete each 
new values of Y, U and V. 60 4YaU:lV block widiin a macro block 477, the time required 
Transf <mn processor 210 also performs matrix transposi- for a single hardware device to execute DCT transform 
tion 736, two-dimensional filter 738, matrix multiplication coding will take 38.4 microseconds which would mean that 
740 and matrix addition 742. These are requh^ since there would only be 36.6 ndcroseconds left for odier time 
whenever motion compensation techniques are applied the demanding tasks such as motion estimation, variable length 
old frame 714 must be filtered before it can be added to die 65 coding and quantization. 

new frame difference 442. Additionally, the Inverse Although pipeline 902 and parallel processing 904 tech- 

(Discrcte Cosine Transform) 485 ou^»ut must be transposed niques can be applied to iini»x>ve system pcrfwmance such 
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as multiple DCT transform pipeline processors 744 can be 
cascaded in parallel as shown in FIG. 7^ this solution is not 
acceptable for the consumer based mass market 

The scalable memory array reconflgurable technique 
reduces the standard OF format 302 to a modified GIF 
fonnat 906 with slightly coarser resolution and yet retain all 
of the integrity of the standard OF format 302 and QOF 
fonnat 304. The scalable memory array has the option to 
choose between the OF f<M3nat 302 or QdF fonnat 304. 

The modified GIF format 906 provides a 288hxl92v 
resolution 908 and the modified QOF format 907 provides 
a 144hx96v resolution 910. This provides dose to the 
original GIF and QOF 302 and 304 quality respectively and 
also maintains the 4:1:1 integrity of the YUV signal 471. 
Each OF format 302 will still retain twelve (12) groups of 
blocks 482 and each QOF format 151 will still maintain 
three (3) groups of block 482. The macro blocks 477 and 
pixel 912 format will remain the same. The only difference 
is that each group of block 482 wtU now consist of 18 macro 
blocks (9hx2V) while the original OF format 302 gxoap of 
blocks consisted of 33 macro blocks (llhxSv). 

This is accomplished during the input and output color 
conversion process in thst COR 601 image 916 input which 
consists of 720hx480v resolution can be downsampled (5:2) 
918 to the 288hxl92v Y resolution and further down- 
sampled 5:1 920 to the 144hx96v U, V resolution. At die 
output display, die Y. V can perform 2:5 upsaiiq>ling 922 
for the Y and 1:5 upsampling 924 for the U and V. The 
significance of this modified OF format 908 design is that 
the internal processing performance requirement Is reduced 
by 46% which means we are now allowed to use slower and 
more economical hardware for encoder processing. 
Meanwhile, memory subsystems, such as frame memory 
214 and first-in, first^ut men>ory 428, can employ slower 
memory devices that reduce costs. 

Secondly, scalable monory array 926 permits the further 
scaling down of our modified GIF format 908 to meet either 
{plication requirements or cost production requirements <r 
to simply drop firom a higher resolution format to a coarser 
resolution fonnat to meet the real time and coding require- 
ment As an example, the GIF frame fcHinat could be 
in^lemented at 144hx96v resolution and a QCSP frame 
fonnat in 72hx48v resolution. Goosequently, the mnttimrdia 
assembly 112 can employ the standard CSF fonnat 302 or 
QGIF format 304 when cost and performance are acceptable. 
In other instances, the scalable memory array 926 would be 
adopted so that the GIF and QCJF formats would be adq>ted 
as per the following frame selection cxan^es. 



Mode 


OF 


QCIF 


TYPE 


1 


352h X 288v 


17£h X 144v 


Standard 


2 


288h X i92v 


144b x96v 


Modified 


3 


144b x96v 


72b X4«v 




4 


72hx48v 


S«b x24v 


Modified 


5 


36bx34v 


L8b X 12v 





The scalable memory array also allows the partition of 
frame memory 214 into sections of modified fr^ames to allow 
multiple processes to run in each frame section. As an 
example, a frame memory 214 of 352hx288v size can be 
scaled down to either a single 288hxl92v section; 4 144hx 
98v sections; 16 72hx48v sections: 64 36x24y sections or 
any of the mixed combinations, all of the sections being 
processed in parallcL 

The scalable memory array can also provide remote 
MPEG 240 video playback. Standard MPEG provides four 
times the resolution in^>rovement over the existing CGIR 
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601 standard. Namdy. the standard MPEG 188 can ptovidc 
1440hx960v resolution. The significance is now we are not 
only able to run each memory section as a parallel process, 
but we are also able to provide con^tibility between the 

5 two standards MPEG 240 and H*261 238. Now, the MPEG 
standard 240 designed originaUy only to provide high reso- 
lution motion video playback locally can now be used to 
transmit con^ssed MFGO programs across the network 
employing the widely available H.261 video codec facilities. 

IQ The scalable memory array also enables the user to manage 
and provide tiie remote transmission of MPEG 240 video 
programs employing confcreuce controller 928, store and 
forward 930 and video distribution 932. 

Jt is therefore possible to either downsample a com- 
pressed MPEG frame 240 into one of the modified GIF 
format 908 or siniply send multiple compressed MPEG 
subframes by partition- Fox example, a 144(Hix960v MPEG 
frame 240 can downsanple 5:1 into a 288hxl92v modified 
GIF frame 908 for transmission and decode and upsan4>lc at 

2Q 1:5 to display it at standard MPEG resdutlon at the cofre- 
spending output 

As an example, the following frame fonnats could be 
utilized to interchange between H.261 238 and MPEG 240 
standards. 

25 



30 



Mode 


MFEO 




TYPE 


1 


1440hx960v 


7»hx480v 


Sundsrd MPEG 


2 


nS2h X 768v 


576h X 384v 


Modified MPBG 


3 


576hx3»4v 


288hx 192v 


Modified MPEG 


4 


352hx288v 


176h X 144v 


Standsrd dP/MFBQ 


5 


288b X 192v 


!44hx96v 


Mcxlified OF/MRBG 


6 


144b X 96v 


72hx48v 


Modified CIF/MPEa 


7 


72bx48v 


3aix24v 


Modified OF/MFGO 


8 


36hx24v 


18h X 12v 


Modified C3F/MFEO 



35 

The scalable mem<xy array formats have significance in 
that due to thefr compact size* they become useful in 
r^esenting moving objects in the foregroutkd when the 
background infonnation Is stUl. The background infonna- 

40 tion wotild be pretransmitted during the intra frame coding 
mode 936, while the different moving objects would be 
transmitted during the interframe coding mode 938. 
Depending upon the size of the moving object* the appro- 
priate size of the modified format will be enjoyed. At the 

45 decoder end, the moving objects will be ovcrlaycd witfi the 
still background context to provide nK>tioo sequence. 

The scalable memory airay is particularly suitable to 
progressive encoding of images when band width needs to 
be conserved. The scalable memory airay will choose the 

50 coarser modified GIF format to transmit the initial frames 
and then utilize a larger modified GIF fonnat to send 
subsequent frames such that die complete image sequence 
will gradually be i^gcaded to the original OF quality. 
The scalable memcay array controller performs as a result 

55 of the cooperation between pixel processor 206 and host 
processor 218. Pixel processor 206 is the local host control- 
ler for the video codec and display subsystem 702 and the 
host processor 218 is Oie global host controller for the 
overall system. The pixel processor 206 seives as the bus 

60 master for video bus 422 and host processor 218 serves as 
the bus master for the system bus 418. Both the video bus 
422 and the system bus 418 are system-wide parallel inter- 
connects. Video bus 422 is specifically designed to facilitate 
the high speed video information transfer among subsystem 

65 con^KMients. 

FIO. 10 illustrates the Pixel processor 206 designing to 
meet the flexible performance for various types of popular 
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video coding algorithms such as the MPEG, H.261 or JFEG. 
Meanwhile, pixel processor 206 can abo perfcvm other pixel 
domain-based proprietary methods. While most pixel algo- 
rithms arc either inter 936 or intra 938 frame coding, tibe 
CCnT and ISO standard algorithms (MTOG, JPEG and 5 
H.261) are transformed domain coding methods raiploying 
fast DCr implementation and inter &ame differencing tech- 
niques. Additionally, MPEG and H.261 also ^ly motion 
compensation techniques. 

The pixel processor 206 is equipped with a 24 bit address lo 
line 1002 to permit it to access 16 mega bytes of program 
memory. The program memory can further be partitioned 
into separate segments with each segment designated for a 
specific coding algorithm. Since pixel processor 306 is 
micro-programmable, it is relatively easy to update the is 
changes while MPEG 240, H.261 238 and JPEG 244 stan- 
dards arc still evolving. 

The pixel processor 206 is also designed with parallel 
processing in mind. The micro programmable architecture 
allows multiple pixel processors 206 to couple over video 20 
bus 420 to provide concurrent program execution for an 
extremely high throughput. This will allow each pixel pro- 
cessor 206 to be dedicated to a coder 1008 function or a 
decoder 1010 function. If 6 pixel processors 206 are 
eiiq)loyed. this will allow the concurrent execution of an 25 
entire macro block 4T7. Similariy, the multiplicity of pixel 
processors depending upon cost and size could pennit the 
process of an entire group of block 482 simultaneously. 

The choice of host processor 218 is somewhat critic^ in 
that it must be able to provide an interface witii the external 30 
host 1006, it must be able to execute the popular DOS 491 
or UNIX program 490 such as WOTd processing spread 
sheet programs and it must be economicaL A suggested 
choice is intd 80286 or 80386 microprocessors. These 
provide a convenient bus interface with ttie AT bus which 35 
has sufficient bus band width to be used as the system bus 
418 of the system. The aforesaid micro-processors also 
provide compatibility with a wide variety of DOS 491 based 
software q>pUcation programs. Additionally, the small com- 
putcr system interface 488 is readily available and capable 40 
of providing high speed interface between the internal 
system bus and the external host 1006. 

FIG. 11 is a schematic illustration oS motion processor 
208 subsystems. Confonning to one cf the H^61 coding 
options, motion processor 208 is designed to idei^ify and 45 
specify a motion vector 1102 for each macro block 477 
within the existing luminance (Y) &ame 474. The motion 
vector 1102 for the color difference for (U, V) &ames 473 
can then be derived as either 50% or the truncated integcr 
valuc of the Y frame. The principle is that for each 16h xl6v so 
source macro block 1108, the surrounding 48hx48v area 
1106 of iqxlated new fr^me 712 will be needed to be 
searched and cooD$>ared. The new macro block 477 having 
the least distortion will be identified as the destination macro 
block 1104 and the distance between the source and desti- 55 
nation macro block will be defined as the motion vector 

uoz 

The direct implementation of motion processor 208 
requires that for each of the four bIo<±s 1109 residing within 
the old source macro block 1108 of the existing frame, the 6C 
corresponding destination macro block 1104 centered within 
the new frame must be identified. Therefore, every 
corresponding, surrounding 6hx6v area 1106 of blocks in the 
new frame must be searched and compared with .the old 
macro block reference in order to derive the best match with 6! 
least distortion. This approach will require 589, 824 cydcs 
of search and compare operatioos. Provided the search and 
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compare operations can be fully pipelined, an instruction 
cycle time of 0.13 nano seconds is still required which is too 
time-consuming for the 75 microsecond per macro block 
real time requirement at 30 ^ updates. 

In order to meet such real time perfonnanoe requirements, 
the motion processor 208 nmst employ parallel i^ocessing 
and multi-processing techniques, llie multimedia assembly 
112 incorporates a fine grain, tightly coupled, parallel pixel 
processor architecture 1112 which provides faster speed and 
better results. This is accomplished by partitioning existing 
macro block 477 into 4 8x8 blocks 1109. Four parallel 
processing arrays 1116 consisting of 24hx24v processor 
elements are configured into nine (9) regions. These nine 
regions of macro processor elements 1114 arc tightly 
coupled together. Each region of the existing frame can have 
direct intcrcoimcction and simultaneous access to its eight 
(8) nearest neighboring regions from the correspondittg new 
frame. Each region ot macro processing elements 1114 is 
designated to perform various types of pixel domain pro- 
cessing functions for the 8x8 block extracted from the old 
source macro block 1108. 

FIG. 11 illustrates a parallel search method for 8x8 blocks 
residing within die old source macro block 1108. Each can 
conduct simultaneous match and compare operations with 
all of their nine nearest neighboring blocks. The outputs of 
the nine matching operations are first locally stored at the 
craresponding regional pixel processor arrays 1116. They 
are then shifted out and summed at the output accumulator 
1118 and adder circuits 1120. The results are then compared 
using the comparator circuit 1122 to obtain the best matdi. 
The physical distance between the new macro block which 
results in the best match and the old reference macro block 
will be ^lied as the motion vector for the old luminance 
macro block. 

The regional pixel processor array 1116 can be reconfig- 
urable and is designed based upon nine banks of processor 
element arrays 1126. Eadi processor element array 882 
consists of sixty-four processor elements 1128. The nine 
banks of processor element arrays 1126 are interconnected 
through shift registers 1130 and switches 1132. In a three- 
dimensional implementatioD, a vertically-cascaded proces- 
sor anay 1138 crossbar switch array 1134 and shift register 
array 1136 can be in^lemcnted. Additional layer such as 
storage array can be added to provide the additional func- 
tionality. This array will be extremely powerful when multi- 
layered packaging becomes available for the chip level 
modules and intergrated circuit technologies. 

Atwo-dimensicKial pixel processor array 1116 can also be 
designed using nine t>ank5 of processor element arrays 1126 
equipped with per^heral switches 1132 and shift registers 
1130. The switches 1132 can be reconfigurable to guide 
direction about the date of flow where the shift registers 
1130 can transfer data from any processor clement array 
1126 or ii^Hit to any other piocessor element array 1126 or 
output Both switches 1132 and shift registers 1130 are byte 

; wide to facilitate parallel data flow. The processor element 
arrays 1126 were designed based upon an Sx8 array of 
simple processor elements 1128. 

The processor element arrays 1126 are designed for 
interconnection among the processor elements so that recon- 

» figuration can be accomplished to meet different application 
needs. The processor elements 1128 are designed so that 
each can be programmed to execute simple instmctions. 
Each processor element 1128 consists of a simple ALU 1140 
whidi can execute sin^ile instructions such as add, subtract, 

\ load, store, compare, etc 

FIG. 12A illustrates the design example of a program- 
mable logic device 1201 which employs a cellular array 
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logic architecture. This figure is used to demonstrate the tion 1226 caa be performed by i^csctting the threshold value 
functionaUty and physical design of tiic device. The practical 1234 and ttitn shifting and quantisizlng the coiresponding 
size for an NxN array is dependent upon ttie ai^cation transf(rm domain coefficients based upon the zig-zag scan 
requirements and the slate of tiie art of the inq)lcmcnting format at each low, medium and high frequency regions. The 
technology. 5 threshold value can be rqixogramincd to adjust the quanti- 

FIG. 12B iUustrates the practical In^dementation of a ration level. The advantage is that as soon as the input image 
ceUular logic processor element 1204 using a charge couple is detected, sampled and thresholdcd, several pixel domain 
device 970 tcdinology. The objective is to provide an pr^ffoccssing functions, such as frame differencing and 
intergrated image sensor array 1206 with the digital prcpro- motion estimation, can be performed right away. The dif- 
cessing capabilities so that image coding for the macro lO fcrencing macro Wocks will be sent to transform processor 
blocks and pixel domain image coding functions can be 210 to perfonn DCT operation 1224, the outwit of die E>CT 
performed. The other objective is to allow die implcmenta- coefficients can further be reloaded into the processor ele- 
tion of on-chip parallel image sensing and parallel image ment array to perform quantization. When band widUi 
processing 976 utilizing the same or compatible technology. reduction control 260 is required, initial thresholding is 
The Cellular array logic architecture illustrated in FIG. 12B 15 combined with a coarser quantization 1226 level to reduce 
arc useful diat they can inq)lement fine grain, tightiy- the image resolution. When Ac system demands faster 
coupled parallel processing systems. They employ single- performance, multiple parallel processor element arrays can 
instruction-multiple-dato 1209 or multiple-instruction- be cascaded to perform concurrent macro block operations 
multiple-daU 1210 techniques to provide system throu^put such as frame differencing, motion processing and quanti- 
whcre traditional sequential conq)uting fails. 20 zation. 

Many ceUular array processors have been designed in die The advantage of charge couple device tedinology 1202 
past. Most of them employ a processor array which consists is its suitability for image processing, multiplexing, and 
of a matrix of processor elements 1128 and switch arrays sUx&gt operations. This can be done both in the analog and 
1134 which can provide programmable interconnect net- digital domain. Therefore, depending upon the application 
works among the processor elements. These cellular array 25 requirement, botti analog processing 1238, digital process- 
p-ocessors are extremely expensive. ing 1240 and memory functions using ttiese processes 

The design illustrated in FIG. 12B Is based upon a mudi element arrays 1126 can be accomplished, 
sln^lcr architecture, the design being dedicated only to FIG. 13 is a sdiematiciUustration of tiie functional model 
image iwocessing and coding appUcations. The majw objec- architecture in order to simplify die functional proems 
tive is to meet real time performance requirements for macro 30 covered out by die hardware previously discussed. The 
block pixel domain processing functions or motion process- principal functional elements con^irise a band width man- 
£jjg ager 1300, a foraaatter 1302, a pixel-domain-codec encoder 

FIG. 12A is tmpXayed to demonstrate how frame differ- 1304 coupled widi a pixd-domain-codec decoder 1306, a 
encing functions can be paformcd for cadi of die incoming transform-domain-codec encoder 1308 coupled with a 
sub-image macro blocks 477. For illustration, 3x3 array is 35 transfmiwlomaiD-codec decoder 1310, a network-domain- 
used to represent macro block sub^image 477 which, from codec encoder 1312 coi^led with a nctwork-domain-codcc 
die current frame, is first shifted into die processor element; decoder 1314 and a controller 1316. 
die corresponding maao block sul>.image of die previous The band widdi manager 1300 provides band width 
frame 1218 is dien loaded into die processor element and die control capability wherein a two-dimensional band widdi- 
comparison functions are performed between die two macro 40 ovcr-lay-lookup-table (BOLUT) can be constructed to map 
blocks to detect if there is any frame difference. Provided die die spedflc band widdi ranges, i.c.. 2.4 Kbs to 100 Mbs, et 
difference is larger than the preset ttireshold value, die macro aU into selective options of media combinations sudi as 
blocks will be mariccd and die macro block marker 1242 and overlay In die audio, video, text and graphics widi various 
macro block difference 1244 b<^een die two frames will be types of quality and resolution. 

stored in frame memory 214. If tiiere is no difference, die « Additionally, during noisy communication environments, 
current frame macro block vahie 1216 will be deleted and die band widdi manager 1300 function is to constantiy 
die previous frame macro block value 1218 will be used for monitor die network to detect abrupt network band width 
display updates. dianges caused by local line degradation or netWOTk traffic 

If an excessive number of macro blocks 477 are identified congestion. The band width manager 1300 will respond by 
widi frame difference, dien a scene or illumination change 50 adjusting die media combinations to accommodate die avail- 
has occurred and macro block processor 1220 will notify able band width. 

host processor and pixd processor 206 and switch die During stable conmiunication environment, band width 
operation from intcrframc coding 1227 to intraframe coding manager 1300 operates to reconfigure die different band 
1228, The significance is that while incoming images sensed widdis specified by die network i^oviding upgradabiHly and 
from die camera, die specific macro blocks widi the frame 55 paraLellsm for time-sharing. 

differendng can be identified and stared. Consequentiy, in The formatter 1302 communicates with die band widdi 
die inteiframe coding modes 1227, only diose macro blocks manager 1300 to ascertain die band widdi availability for 
477 requiring motion estimation and compensation 1222, incoming or outgoing signds. The formatter translates dus 
transferal coding 1229 or quantization 1226 will be marked external information into an internally-operating format 
and stored in die frzme memory 214 to represent die image fio The scalable memory array reconfigurable technique will 
sequence of die current frame. In die case of scene or reconfigure die intemd processer and frame memoiy struc- 
illumination changes, enough maao blocks will be detected turc pursuant to die directions of die formatter. This allows 
wltii frame differencing diat die system will automatically die external format to be translated into a suitable internal 
switch to intraframe coding mode 1228. fOTmat to provide system compatibility. The scalable- 

FIG. 12B illustrates additional pixel domain processing 65 memory-anray-reconfigurable-tcchniquc (SMART) as dis- 
ftinctions induding low pass filtering 1230; high pass fll- cussed in FIG. 9 is capable of transUting a programmable 
tering 1232 and quantization 1226. The variable quantize- internal format in compUance widi a wide variety of intcr- 
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oatiooal standard and custom video coding algorithms such 
as NfFEG, H^61, JPEG and vector quantization. Fcnnatter 
1302 identifies the transmitting or receiving coding 
algorithms, derives their specific format requirements and if 
these external format requirements are dijOTerent fi'om the 
current internal formats, the formatter reformats die hori- 
zontal and vertical resolution which results in a sq>arate 
internal format which is compatible with the external for- 
mat. These intonal format operations, such as the reduction 
of the horizontal and vertical resolution, are perfonned by 
employing interpolation and downsampling techniques or 
upsanq)ling techniques. The formatter 1302 also c<Hnmimi- 
cates widi the fi^mc memory so that tibe £rame memory is 
aware of the internal format to be stored. This allows the 
formatter 1302 in conjunction with the scalable memory 
array configurable technique to f CEimulate a scalable proces- 
sor and frame memory architecture so that the internal 
processor and frame memory can be continually adjusted in 
order to reconfigure or modify a suitable internal format for 
any type of external format either being received or sent by 
the network-domain-codec 1314. 

The oetwcMic-domain-codec encoder 1312 and decoder 
1314 are used to provide line coding and decoding functions. 
Network domain codec decoder 1314 would receive net- 
work transmissions via its front end transceiver 1320. It 
would then perform protocol procedures 1322, network 
communication procedures 1324, variable length coding 
1326, run length coding 1328 and filtering 1330. The result- 
ant transform cocffideots and pixel data will then be for- 
warded to either pixel-domain-codec decoder 1306 or 
transform-domain-codec decoder 1310. The network- 
domain-codec encoder 1312 would receive encoded pixel 
data or transfonn coefficients from the other encoders and 
convert them Into serial codes for network transmission 
performing functions similar to the network domain codec 
decoder 1314. Simultaneously, band width manager 1300 
will interface with encoder 1312 and decoder 1314 to 
exchange protocol control and i^lications information 
regarding band width availability. 

The pixcl-domain-codec encoder 1304 and decoder 1306 
are designed for custom coding algorithms such as vector 
quantization, pixel domain operations for the DCT trans- 
form based standard coding algorithms such as MFEG, et ai, 
poxel domain operations for motion compensation and image 
post-processing functions and analysis and preprocessing 
techniques for video coding. Thus, the pixel-domain-codec 
provides for pixel domain preprocessing 1332, pixel domain 
coding 1334, image processing 1336, color space conversion 
133S, pixel interpolation 1340, vector quantization 1342 and 
color lookup mapping 1344. 

The transform-domaiD-codec encoder 1308 and decoder 
1310 are specifically designed for forward and inverse 
transformation operations required by the international stan- 
dard coding algcHithms such as MTOG, et al. T^sfoim- 
domain-codec encoder and decoder 1308 and 1310 also 
provide forward and inverse transfoim-based operations 
such as Hart transform and Hadamard transform. 
Additionally, generic matrix operations and post-matrix 
operations, such as scan convcrsiOD, quantization and nor- 
malization techniques, are performed by the transfonn- 
domain-codec. 

The controller 1316 conqaised of dther a single or 
plurality of local host processors which manage ti^e instruc- 
tion sequencing and system control functions for data 
transfer, memory management Input/ou^ut interfacing and 
processor pipelining. 

Id fig. 4, we demonstrated a host processor used to 
manage the communications pipeline, the network domain 
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codec and the system memory. It also perfcsmed general 
administrative tasks and controlled the system bus and 
access to other subsystem buses while communicating with 
the band width manager 1300. 

A second controller is a single or plurality of pixel 
processors used to manage the video pipeline, &e scalable 
memory array reconfigurable technique, firame memories, 
formatters and display t^ocessing. Additionally, the pixel 
processor is used to perform pixel-domain-codec encoding 
and decoding functions and can be used in mult4>les in order 
to facilitate macro block and group of block processing. 
Similarly, a single or plurality of transfc»m processors can 
be cmi^oyed as coprocessor for the pixel processors, in 
performittg transform-domain-codec encoding and decoding 
functions. 

All oetwcxk transmissions or receiving functions would 
first pass through the netwoik-domain-codec and then be 
directed to the pixel-domain-codec or transform-domain- 
codec after suitable formatting. The media information 
could then be displayed via the pixel-domain-codec decoder 
1306. Origination signals from either storage, camera, TV or 
CD would be subjected to frame differencing 1364 and 
fran^ image capture 1366 before being encoded by pixel- 
domain-codec encoder 1304. These origination signals 
could then be transmitted via network-domain-codec, 
encoder 1312 dependent upon the band width manager 1300 
and controller 1360 mooitoiing of band width availability. 

While the invention has been described with reference to 
its preferred embodiment thereof, it wiU be appreciated by 
those of ordinary skill in the art that various changes can be 
made in die process and ^qjparatus without departing from 
the basic spirit and scope of the invention. 

I daim: 

1. A controller i^paratus for executing a plurality of 
control functions for communication of multimedia articles 
including voice, high quality audio, text, still image, motion 
video and animated graphics between multimedia equip- 
ment and communications networks, selectively providing 
receiving functions, retrieving functions, and transmitting 
functions, wherein said controller is con^;>atible wish rmil- 
tipie band widths and algorithmic coded signals, said con- 
troller providing a scalable architecture for communication 
of multimedia data, formatting from internal format to 
external format in compliance with any selective one of a 
plurality of international standardized and customized coded 
algoridims, and band width determination of available dis- 
play band widths according to frame rate and available 
transmission band widths, said controller conqirising: 
means for receipt of an external network signal, said 

means comprising a network domain decoder, 
means for receipt of a local origination signal; and means 
for image processing of said origination signal for 
storage cocting simulation, and transmission; 
a first control noeans and format means fcsr selectively 
formatting the f<»mat of said extx^nal network signal 
and local origination signal to a compatible first inter- 
nal format in compliance with internal processing and 
memory capabilities; 
means for decoding said external network signal accord- 
ing to said first internal format, said means selectively 
con^iising a transform domain decoder alone and in 
combination with a pixel domain decoder; 
a second oontrd means and format means for selectivdy 
formatting said external network signal and said origi- 
nation signal to a second internal format in compliance 
with a selective plurality of coding algorithms for 



03/01/2004, EAST Version: 1.4.1 



5,706,290 



21 



22 



processing, storage and traQsmissioii, said means fur- 
ther coroprising an encoding means which comprises a 
pixel domain cacodcr alone and in combination widi a 
transfonn domain encoder for encoding said external 
network signal and said local origination signal; 
means fct reconfiguring the laemory means fca* selec- 
tively associating with said apparatus encoders and 
decoders for directing said first and second format 
means; 

band width management means, in communicati(Hi with 
said second format means and said control means, fcs 
directing available run time transmission band widths 
and appropriate comjHcssion ratios for selectively for- 
matting said second internal format to external trans- 
mission format in compliance with available transmis- 
sion band width range, said band width management 
means continuously moderating and correcting said 
selective formatting of said second internal format to 
said second external transmission format based on 
transmission band width availability; 

means for transmission of said internally formatted exter- 
nal network signal and local origination signal accord- 
ing to said external transmission format, said means 
comprising a network domain encoder. 

2. A contrdler ^>paratus in accordance with daim 1 
wherein said network domain decoder for receipt of said 
extcmal network signal selectively conqsrises a front and 
transceiver; protocol processor; n^ork communications 
processor, variable length decoding processon run length 
decoding processor; and filtering processor for sampling 
said external network signal, said network domain decoder 
in communication with said first control means and said 
format means to convert the horizontal and vertical resolu- 
tion of said external network signal to an internally-defined 
file format size for reducing said internal real time video 
processing and frame memory updating requirements. 

3. A controller ^>paratus in accordance with claim 1 
wherein said second controlling means and said second 
formatting means, in communication with said network 
domain encoder and decoder selectively codes and decodes 
said Incoming signal based upon Che availability of custom- 
ized and international standardized algorithms and selec- 
tively performs motion compensation processes and modon 
simulation processes to improve quality of said motion 
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means for performing integrated parallel sensing and 
storage wherein a plurality of input frames for said 
local origination signal can be sensitized, digitized and 
stored; and 

means for performing detection of instantaneous scene 
changes wherein only the corresponding frame will be 
registered as intraframe, wherein all other fiames will 
be registered as interfirames, said detection means fur- 
ther con^irising means for moderating the threshold 
level for proper determination of said scene changes; 
and 

means for perf(xming in analog and digital fonns parallel 
image preprocessing for said interfirames whereby each 
interframe image can be retrieved and compared to 
develop a plurality of corresponding fi-ame differencing 
articles whereby only said frame differencing articles 
are required for further processing and encoding 
including motion estimation, discrete cosine transform, 
quantization, and hufiman-like variable length coding. 

6. A controller ^aratus in accordance with claim 5 
wherein said means for image preprocessing for each said 
inteifrwe further comprise a means for motion estimation, 
motion simulation and motioo compensation operations on 
motion video articles coroprising: 

means to perform edge detection operations for each of 
said firame differencing articles wherein a selective 
plurality of detected edge articles are derived to iden- 
tify said motion video articles during motion estimation 
and compensation operations; 

means to perform feature extraction operations for each of 
said fr^amc differencing articles wbereiD said extracted 
specific features for said frame differencing articles can 
identiiy from said motion video artides during said 
motion estimation process; and 

means to perform forward, backward and random refer- 
ence of selective inteiframes eii^)loying said selective 
features and said selective detected edges for said 
frame differencing artides. 

7. A controller apparatus as set forth in claim 5 wherein 
said motion video artides indude a plurality of moving 
foreground pixels overlaying can be referenced using trans- 
mission of a single or plurality of preceding intraframes 
wherein said small moving foreground pixels for subsequent 



video aitide, said processes indude pattern match, edge 45 plurality of interframes can be referenced using frame 



detection, image enhancement, color mapping and pixd 
iateipolation of said external netwoik and local origination 
signal 

4. A controller apparatus in accordance widi daim 1 
wherein external network signal is selectively transferred to 
said transform domain decoder and said pixel domain 
decoder for decoding; said transform domain decoder selec- 
tively providing denormaiization, dequantization, scan 
conversion* matrix operation, inverse Haar transform, 
inverse Hadamard transform and inverse discrete cosine 
transforms on said external network signal and said pixd 
domain decoder selectivdy performing color mapping, vec- 
tor dequantization, pixel extrapolation, color space 
conversion, image processing, pixel domain decoding, and 
pixel domain preprocessing on said external network signal 
for formatting of said external network signal and conver- 
sion to said first intcmal fonoat for cither storage processing, 
di^lay or transmission. 

5. A controller apparatus in accordance with claim 1 
wherein said means for image preprocessing of said local 
originadon signal comprises an integrated circuit subsystem 
further comprising: 
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differencing techniques wherein each group of said moving 
foreground pixels are characterized employing said frame 
differencing of a plurality of said detected edges and a 
plurality of said extracted features. 

8. A controller apparatus in accordance widi claim 7 
wherein a motion video artide can be applied with one or 
more pattern searching and recognition algorithms to derive 
alternative reference motion vectors wherein said motion 
vectors can forward, backward and random reference said 
motion video article during said interframe sequencing said 
motion vector can be represented by the interframe displace- 
ment of symbolic representation further comprising a plu- 
rality of detected edges, fi:amc differences and extracted 
features for each cotresponding motion video article, said 
symbolic representation and alternate motion vector can be 
applied for transmission simulation, coding, storage and 
processing of said motion video articles. 

9. A controller apparatus in accordance wiOi claim 5, 
wherein said means for interframe image preprocessing 
further comprises parallel processor means for perfotning 
pixd level operations for an nxn pixel block sub-image, said 
means comprising an m>aa array of said parallel processor 
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dements wherein n and m are integers and said parallel 
processor means can retrieve, decode and execute instruc- 
tions for pixel input data from intanal memory; and inter- 
connection means fCM* selective communication between 
cadi of said parallel processors whereby said pixd data 
inputs and Incoming control messages can be received from 
said selected parallel processors and said pixd data ou^ts 
and outgoing messages can be broadcast to said sdectcd 
paralld processing neighbors. 

10. A controller ^paiatus in accordance with daim 5 
wherein said means for interftame image preprocessing 
further comprises a processor array configured as an nxn 
cellular logic array j^ocessor where n is tyiHcally of value in 
arange of between 8 and 32 wherein said cellular logic array 
processor can be employed to pcrfonn a plurality of block 
processing operations for said nxn pixel Mock of sub-image, 
said processor array fui^cr con^irising: 
a means for &ame differencing between interfirames 
wherein a group <rf nxn difference blocks are identified, 
registacd and stored; 
a means for template matching and artide recognition 
op^ations for eadi of said nxo pixel blocks of said 
sub-image article extractiiig from said input image and 
comparing with sdective nxn reference data stcared 
previously in memory; 
a means for determination of threshold level for selective 
matching said sections of said sub-image artide to 
identify a motion vector artide; 
a means for motion estimation operation which can be 
applied to each of said nxn sub-image blocks for 
comparison between said previously-displayed frame 
article and said current frame artide, said nxn sub- 
image performing matdiing functions at said frame 
differencing locations for generating horizontal and 
vertical distance for representing said motion vector, 
a means tor edge detection for each of said nxn blocks oi 
said sub-imAge and means for edge enhancement for 
each of said decon^cssed image blocks prior to 
display, said edge detection for deriving said motion 
vector; 

a means for feature extraction and con^arisoo of said 

previously-displayed frame witti said current frame; 
Hadamard transform means for each of said nxn Wocks of 
said sub-image wherein eadi of said nxn blacks is 
represented by its paxd domain transform coeffident 
multiplied by the nxn Hadamard coeffident matrix 
thereby icdudng transmission band width through 
pixel domain band width compression; and 
frequency domain transform means Induding discrete 
cosine transform means wherein each of an 8x8 frame 
differencing sub-image block can be nmltiplied by an 
8x8 transform coefficient block to derive 8x8 blocks of 
frame differences identified by said frequency domain 
transform coefBcients* 
11. A contiollCT ^)paratus in accordance with daim 1 
wherein said second control means and second format 
means for sdectivdy formatting said internal format further 
courses a means for template matching pattern recogni- 
tion aiKl motion estimation and con^)cnsation, comprising: 
means for extracting cadi of the nxn luminance pixd 

blocks from said previous reference intcrframc; and 
means for performing tcraplaXe matching operation of said 
nxn reference block within a mxm window of sur- 
rounding pixel blocks residing in said current inter- 
frame wherein m and n arc integers and m is greater 
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than n, said template matching generatiDg a motion 

vector for said pixd block in the event of a match; 
means for determining and adjusting said mxm window 

size in the event of no match; 
means for performing pattern recognition on said adjusted 

nxn window sub-image. 
IZ. A controller apparams in accordance with daim 1 
wherein said image prqjrocessing ^aratus further com- 
prises a means for identifying an estimating randomly- 
generated signal noise embedded within the input image 
artide; and means for subtracting said randomly-generated 
signal noise from said original input image artide by means 
of a first sensor and a second sensor, said first sensor 
sensitizing image and noise and said second sensor sensi- 
tizing noise for conq)arison of said sensors and subtraction 
of said noise signal. 

13. A controller ai^uratus in accordance with daim 10 
wherein said article means for template matching further 
comprises: 

means for retaining color representation of each signifi- 
cant object of said frame; 

means for storing said coic^ rqsresentation; 

means for retrievii^ said color representation and pa- 
fcrming pel domain transform operations; 

means for converting said color representation from said 
pel domain to said frequency domain in order to 
perform additional frequency domain processing 
noeans such as discrete corine transform and fast fouricr 
transfomt 

14. A controller apparatus in accordance with daim 1 
wherein said band width management means comprises a 
means for generating a network transmission request signal 
for the transmission band width requirement for outbound 
transmission nmltimedia artides; 

means for recdving an acknowledgement signal from an 
extension network c(»xtroller with respect to transmis- 
sion band width requirement; 
means for comparing said acknowledgement signal with 
said outbound transmission signal to determine suffl- 
ciency of transmission band width; 
means for adjusting said oudxmnd transmission signal to 

accommodate said band width availability; 
means for conqaressing and deconqiressing said outbound 

transmission signal; 
means for transmitting said outbound transmission signal; 
means for continuous monitoring said outbound transmis- 
sion and said band width availability and sdectivdy 
adjusting said outbound transmission signal to oonfcHm 
to said availability of said band width during said 
transmission. 

15. A controller apparatus in accordance with daim 1 
wherein said means for selectively reconfiguring memcB^ 
conqjrises: 

means for definition of a default internal file format and 
size based upon coder processing and frame memoiy 
system throu^put; 
means for receipt of an external algorithmic coded signal 
during network int>ound stage, said means comprising 
a network domain decoder; 
means forrecdpt of a local origination signal, said means 

conqvising a capture processor; 
means for identifying and recdving artide types and file 
size during said network inbound stage, said means 
con^vising a host processor and its interface to said 
capture processor or said network decoder; 
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means for providing adequate downsan9>ling ratio, 
wherein said receiving article can be coofonned and 
reduced to said predefined internal file format and size; 
said means comprising a rcconiiguration unit and a 
scaler circuit; 

means for manipulatioD of said Internal file articles, said 
means conqirising an article-oriented instnictioo set, 
and means for staring retrieving, decoding, and execut- 
ing said instruction set by said host processor and said 
pixel processor; 

means for readjustment of said intanal lilc format and 
size in oonfonnance with band width availability, said 
means conqrising a band width management unit in 
coimnunication with said scaler circuit, reconfiguration 
unit and said host processor; 

means for adjustment of said internal file fonnat and size 
during intei&ame coding modes, wherein a plurality of 
motloa video foreground articles can be specified to 
proper size, and wherein static background articles can 
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a means for receiving a transmit acknowledge signal from 
a networic controller and destination receiver indicating 
said band width availability; 

a means for relaying a band width availability signal to a 
band width management unit for comparison of band 
width availability to transmission signal; 

a means for generating an aUemate compression ratio for 
said transmission signal compatible with said band 
width availability; 

a means for in^ementing said alternate compression 
ratio dtfough a host or encoding processor for further 
compression of said multimedia article to meet said 
band width availability; 

a means fOT transmitting said mulfi media article on said 
available band width; and 

a means for monitoring said multimedia article transmis- 
sion in said band width availability during transmis- 
sion. 

18. An ^aratus for the continuous management and 
control of transmission band width availability in accor- 



be pretransnoitted during intrafiramc coding modes, said 20 ^^^^ u wherein said means for generating said 



means comprising a frame differencing drcuit and 
motion processor interfaced with said pixel processor; 
and 

means for providing upsampling ratio during 
transmission, wherein said transmitting article can be 
conformed and expanded to a selectively-desired file 
format and size, said means comprising said reconfigu* 
ration unit, and said scaler circuit interfaced with said 
band width managemmt unit 

16. A controller ^jparatus in accordance with daim 1 
wherein said first control nkeans and second control means 
communicate across the network with external devices 
including pagers, remote control means and host conqtuters 
for the selective exchange of comomand, data, status and 
control inforroation with said external devices wherein said 
external devices perform selective multimedia applications 
for said oootroUer, said communication means between said 
controlling means and said external devices comprising: 

means for providing conmiand layer protocol control 
functions in the form of predefined commands selec* 
tively interpreted by said control means; 

means for updating and rearranging said predefined com- 
mands; 

means for prioritizing said predefined commands; 
means for providing transport layer protocol control func- 
tions; 

means for data transfer and data tracking in cooperation 
with said command layer protocol; and 

means for providing physical layer protocol control func- 
tions for implementafion of initialization termination of 
mechanisms. 

17. An apparatus for the adaptive continuous management 
and control of network traffic condition and band width 
availability for the communication of multimedia articles 
including voice, audio, text, still image, motion video and 
animated gr^hics between multimedia transmitter and 
receivers wherein said a^^aratus is compatible with multq>le 
analog and digital transmission standards, such as analog 
voice grade line, PSDN, basic rate ISDN, primary rate 
ISDN/ri, LAN, and FDDL the ^aratus having a band 
width detection means determining available display band 
widths in accordance with frame rate and run time trans- 
mission band width, said apparatus con^irising: 

a means for generating a transmission request signal 
provided with message retaining transmission band 
width for an outbound multimedia article; 
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transmission request signal and said means for generating an 
ahemodve comix^ssion ratio further comprise: 
a means for sensitizing outbound line condition changes 
whereby outbound transmission bit rate is adjusted to 
provide sufGcient composition of said multimedia 
article groups, said means for sensitizing said outbound 
line condition conqvising a selection of a plurality of 
quality levels for digitally-con^Hcssed audio articles; a 
selection of a plurality of quality levels for digitally- 
compressed motion video articles; a selection of a 
plurality of quality levels for digitally-compressed still 
images; a selection of a plurality of frainc updating 
rates for digitaUy-compressed motion video articles; a 
selection of phirality of quality levels for digitally- 
coded animated bit mapped or vector gcssptdcs articles; 
and a selection of a i^urality cf options for composites 
of said digital, audio, image, video and animated graph- 
ics articles; and 
a means for selectively monitoring and adjusting said rate 
requirements for said outbound nmltimedia articles. 

19. An apparatus in accordance with claim 17 for the 
continuous management and control of said band width 
availability, said apparatus further conqsrising: 

a plurality of predefined program sequences for a plurality 
of transmission line conditions whereby each of said 
program sequences oonqirises a specific group of zmil- 
timedia articles, predefined for said transmission band 
width availability; and 

a means to store said predefined program sequence and 
reference said predefined program sequence during line 
condition changes. 

20. A method for tiie adaptive continuous management 
and control of network traffic condition and band width 
availability having a band width detection means detcmun- 
ing available display band widths in accordance with frame 
rate and run time transmission band width, for the commu- 
nication of multinv^ift articles including voice, audio, text, 
still image, motion video and animated graphics between 
multimedia transmitta and receivers wherel^ said i^para* 
tus is compatible with multiple analog and digital transmis- 
sion standards such as analog voice grade line, PSDN, basic 
rate ISDN, primary rate ISDN/Tl, LAN, and FDDI, tiic 
method coaoFsislng: 

(a) generating a transmission request signal provided with 
message band width requirement for outbound multi- 
media articles; 
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(b) receiving a transmit adaiowledge signal from an can be fonnatted and sized, said means comprising said 
external network controller and destinatitm receiver reconfiguration imit, and said scaler circuit in intoface 
indicating said band width availability; with a band width management unit 

(c) relaying said band width availabiUty to a band width ^ ^ scalable, reconfigurablc memory apparatus for 
management unit for comparison vwth band width of 5 executing a plurality oflF preprocessing and post-|H-oocssing 
said outbound multimedia articles; functions for video coding in Accordance with daim 21, 

further conmnsmg: 

(d) genexa^g *?.^teniaUve compression ratio for said anintegiatcdparallelprocessor and ston»ge array whereby 
outb<mnd midtimeda articles compatible with said ^ pitmdity of frames of said input image can 1^ 
available band width; sensitized, digitized and stored; 

(e) relaying said alternative compression ratio to a host a: ^ parallel image ptcproccssor array for performing real 
encoding processor for further compression of said ^ encoding opcratioDS, said preprocessor 
outbound multimedia articles in conf<Hmance with said ^jj-ay including a frame differencing operation whereby 
band width availability; i^g^ encoding operations, said input images 

(f) transmitting said outbound multiroedia aiticies in can be sensitized and compared to develop frame 
conformance with said transmission band width avail- differencing articles whereby said frame differencing 
ability; articles can be further processed for motion estimation, 

(g) monitodring said band width availat^ty and said discrete cosine transfoim, quantization, and huf&nan 
transmission of said outbound xmiltimedia articles dur- variable length coding whereby said motion estimatiott 
ing said transmission; 20 and motion compensation operations conqirise an edge 

(h) generating alternative compression ratios during trans- detection operation fw each of said frame differencing 

mission to adi^ to changes in said band width avaQ- wbCTcby a plurality of detected edge articles are 

^^j^. derived to identify said motion video objects during 

... ^. , ^ ru\ -J said motion estimation and connjcnsation operations; 

(t) repeating steps (a) through (h) as required. *^ 

21. A scalable, reconfigutable nxonory apparatus for ^ ^ . . ^ ^ ^ 

executing a pluraHty of preprocessing and post-processing " ^J^'^^T^^ "^^^ 

functions for the communication and storage of multimedia «^ difierencing articles can 

articles including voice, audio, text, still image, motion / ^ . , 

video, and anim^ed giaphics between sdectivTmultimedia ^ / scalable reconfigurablc memory apparatus for 

transmitters and recewSs. said mm^ apparatus compat- » f<«!tmg a pluratey of preprocessing and post-processing 

ible with multiple standard or cusUmuzed coding algortdh- f""*^^™^ accordance with cbum 22 wheran said inpirt 

mic signals including H.261, MPEG, JPEG. EOTV or images comprise moymg foregroimd paeb 

Hirrv, said arawtatus coDMwising: ™*ec background pixels whereby saxd bad^round pads 

may be ^aracterized singly during interframe coding, 
means for definition of a default internal file fonnat and 3, ^^^^ transmission, and said moving foreground 
size based upon coder j^ocessmg and frame memory ^^^^ ^ ^jj^^g interframe differencing 
system tiiroughput; technique in said motion estimation and con5>cnsation tech- 
means for rece^ of an external network algorithmic nique \^^ereby said parallel processor performs pixel level 
coded signal, said means com(Hising a network domain operations for an nxn pixel block sub-image, said parallel 
decoder, 40 processor conqirising an mxm array of pualiel processor 
means for receipt of a locally-originated signaL said elements whereby n and m are integers of equal or unequal 

means comprising a capture processor; value, 

means for identifying and receiving said m 'iT^i mr^ i> ^ scalable, reconfigurablc meoKHy apparatus for 

article and fUe size, said means comprising a host executing a plurality of preprocessing and post-processing 

processor selectively interfaced witii said capture pro- ^5 functions in accordance with claim 23 wherein said parallel 

cesser and n^ork domain decoder processor elements can retrieve, decode and execute instruc- 

means for downsampling whaeln said imiltimedia aitick P^f, ^ ^ intern^ buffex or in 

can be confonned and reduced to a predetennincd >»«ghboniig parafld processor elements buffer, said p«cl 

internal file format and size, said means conmrising a data can be st«ed internally and in naghboruig 

reconfiguration unit and scaler cirout; » ^ .f°, *"»«~°»«^«" proff«m. 

- 1^ ^ .J . * 1 £1 £ . -J mlng each of said parallel processors to a selective group of 

means for mampuhmon of said internal file fonnat. said paraUd proc^sors whereby pixel da<I in^ 

means comprising said host processor and a p«el ^fc^^U ^Ksagercan be received fr^ said selected 

processor, parallel processing neighbors and pixel data outputs and 

means for adjustment of said internal file format and size 35 outgoing messages can be transmitted to said selected group 

to confOTm to band width availability, said means parallel processor neighbOTS. 

comprising a band width managenient unit in commu- jS. A scalable, reconfigurablc memory apparatus for 

nication with said scalCT circuit, said reconfiguration executing a plurality of preprocessing and post-processing 

unit and said host processw; functions in accordance witti claim 24 wherein said parallel 

means for adjustment of said intonal file format and size processors can be arrayed and configured as an nxn cellular 

during interframe coding, wherein motion video fere- logic array processor wherein n is a value of between 8 and 

ground articles and still background articles are differ- 32 and said cellular logic array ptocessor can perform 

endated and separately transmitted, said means com- processing <^>erations for said nxn pixel blocks of sut>- 

prising a frame differencing circuit, and motion images including said frame differencing operation between 

processor in interface with said pixel {xocessor; 55 ^ current frame and a {»'eviously-displayed frame whereby a 

means for providing upsampling during transmission group of nxn differencing blocks are identified^ registered 

wherein said transmission of said multimedia article and stored; template matching operations for each of said 
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OXD pixel blocks of sub-images, said ten^late matching 
operatioD extractiiig ao input image and comparing with a 
previously-stoired input image thereby deriving a motion 
vector, pattern matdiing; motion estimation ^lied to each 
of said nxn sub-image blocks residing within said 
previously-displayed frame; edge detection for eadi of said 
nxn blocks; feature extraction; Hadamard transfonn for each 
of said nxni>locks of sub-images whereby each said nxn 
blocks can be rqiresented by a pixel domain transform 
ooe£ELcient and a Hadamard coefficient matrix; and a disoetc 
cosine transform operation. 

Z6. A scalable, rcconfigurable memory array method for 
executing a plurality of preprocessing and post-processing 
functions fc? the communication and storage of multimedia 
articles including voice, high quality audio, text, still image, 
motion video and animated graphics between multimedia 
transmitters and receivers, whereby said method is compat- 
ible with multiple standard customized coding algorithmic 
signals including H^61> MPEG, JPEG, EDTV or HDTV, 
wherein said method provides a scalable architecture for 
conQmunication of multimedia data, formatting from internal 
format to external fcmiat in compliance with any selective 
one of a plurality of interaational standardized and custom- 
ized coded algorithms, and band width determination of 
available display band widths according to frame rate and 
available transmission band widths, said method comprises: 

defining of a default internal fiic format and size based 
upon coder processing and frame memory system 
throughput; 

selectively receiving an external network algorithmic 

coded signal and local <Higination signal; 
identifying the multimedia article signal and size; 
downsampling the received multimedia article to a pre- 
defined internal file format and size; 
maidpulating said internal file fcarmat to conform to 

transmission band width availability; 
distinguishing between motion video foreground articles 
and still background articles during Interframe coding 
and intraframe coding modes; 
upsamj^g said internal file format to selectively desired 
fcMrmat and size for transmission in accordance with 
transmission band width availability; 
distinguishing between motion video foreground articles 
and still background articles during interframe coding 
and intraframe coding modes; 
upsanq>ling said internal file format to selectively desired 
format and size for transmission in accordance witii 
transmission band width availability. 
27. A scalable, reoonfigurable memory airay method in 
accordance with claim 26 wherein said method for manipu- 
lating said internal file format to conform to band width 
availability and distinguishing between motion video fore- 
ground articles and still background articles during inter- 
frame coding and intraframe coding modes further com- 
prises: 

differentiating between still image background pixels and 
moving foreground pixels; 

performing frame differentiating techniques on an nxn 
pixel block sub-image by means of an mxm array of 
parallel processors whereby d and m are integers of 
equal or unequal value; 

comparing current frame images to previous frame 
images and template matching said images; 

deriving a motion vector for eadi of said frame differ- 
ences; 
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performing pattern matching, motion estimation, edge 
detection, feature extraction and Hadamard transform, 
for each of said nxn blocks of sub-images; 

representing each of said nxn blocks of sub-images by a 
pixel domain transform coefficient, Hadamard coeffi- 
cient matrix and discrete cosine transform. 

28. A method for die application of image preprocessing 
techniques fcr the interframe coding of multimedia articles 
induding voice, audio, text, still image, iDotion video and 
animated graphics between multimedia transmitters and 
receivas, wherd^y said method can be integrated witii an 
image capturer circuit and band width management circuit, 
said mctiiod comprising: 

(a) cq)turing a local origination still image in accordance 
with frame update requirements established by a host 
processor; 

(b) capturing and staring the complete image frame 
during the intraframe codiog mode; 

(c) coixq)aring and differencing said present captured 
frame to a previous c^tured frame during the inter- 
frame coding mode; 

(d) storing the frame differences between said present 
frame and said previous frame in bloclcs and macro 
blocks; 

(e) registering and coding said frame differencing blocks 
ot said frame differencing macro blocks into a frame 
differencing bit map; 

(f) converting color space from analog RGB or NTSC 
format into a digital RGB or YUV format; 

(g) retrieving the luminance macro blocks according to 
said frame differencing bit map; 

(h) deriving the appropriate motion vector for each of said 
macro blocks through ccnnparison and selection of the 
minimum distortion among adjacent macro blocks; 

(i) reducing from an external file format to an internally- 
defined format in accordance with a downsampling 
ratio established by said host processor, the reconfigu- 
ration unit and the scaler; 

(j) translating said frame differencing bit map to conform 
with said internally-defined format; 

(k) performing pixel domain and transform domain cod- 
ing operations for said frame differencing macro blocks 
for said Y, U and V format; 

(1) enlarging from said internal file format to an external 
file format aocordiog to an appropriate downsanqiling 
ratio as estaUished by said host processor, reconfigu- 
ration unit and scaler, 

(m) adding said frame difference to said previous frame in 
order to display and update said current frame; 

(n) repeating steps (a) through (m) as required; 

(o) repeating steps (a) through (n) for live video sequence 
as required. 

29. A method for the ai^licatioo of frame diffaencing 
techniques, pixel interpolation techniques and annealing 
techniques for the conq)ensatiott of live motion video 
articles which may include voice, audio, text, still image and 
animated graphics, said motion compensation method pro- 
vides a scalable architecture for oomnmnication of multi- 
media data, formatting from internal format to external 
format in compliance with any selective one of a plurality of 
international standardized and customized coded 
algorithms, and band width determination of available dis- 
play band widths according to frame rate and available 
transmission band widths, said method con^sing: 
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(a) capturing the scene of a modon video artide through 
frame diffacDcing techniques; 

(b) representing the motion video profile of said motion 
video article in an article-oriented instruction set for- 
mat; 

(c) defining said motion video profile of said motion video 
artide by individual object and relative movement 
assodation; 

(d) augmenting said motion video profile of said motion 
video artide by means of rotation, shifting and shuf- 
fling operations; 

(e) reformatting said motion video profile of said motion 
video artide to an intanal file fonnat; 

(0 enlarging said motion video aitide by means of 
upsampling and reducing said motion video artide by 
means of downsampling operations; 

(g) overlaying said motion video [Hofilc of said motion 
video artide with said voice, audio, text, still image or 
said animated graphics to construct a coiiq)lete multi- 
media presentation; 

(h) defining said motion video profile of said motion 
video artide; 

(i) interpolating said motion video profile of said motion 
video article through regeneration of absent pixels to 
further refine said motion video article; 

(j) simulating said motion video profile of said motion 
video article through cq)turing and recording selective, 
nonoonsecutive frame references; 
(k) annealing said motion video profile of said motion 
video article through reconstruction of said absent 
pixels, absent motion articles or absent £rames, employ- 
ing graphics animation techniques; 
(1) selectively repeating steps (a) through (k) as required. 
30. A communication system assembly apparatus execut- 
ing a plurality of control, processing, communication and 
interface functions for the conununication and storage of 
multimedia articles including voice, audio, text still image, 
motion video, and animated graphics between multimedia 
ttansmitters and recdvcrs whereby said communication 
system is con^tible with multiple standard or customized 
coding algorithmic signals such as H.261* MPEG, JPEG, 
mxrv or HDTV wherein multiple incompatible video cod- 
ing equipment employing different video coding algOTithms 
can communicate with each other, said conomunication 
system providing a scalable architecture for ccamnunication 
of multimedia data, formatting from internal fonnat to 
external format in compliance with any sdective one of a 
plurality of international standardized and customized coded 
algorithms, and band width determination of available dis- 
play band widths according to frame rate and available 
transmission band widths, said communication system pro- 
viding: 

means for network communication processing, said 
means conqirising a network domain decoder and net- 
work domain encoder, said means can provide control 
interface to a tdecomraunication network or micro- 
wave network, said means for network communication 
processing induding receiving functions for serial 
compressed video bit stream; 

means for transmission processing, said means compris- 
ing a formattff to transform between a CJP/QCSP 
format and sdective CCnT H.261, NffiEG <s JPEG 
formats, said means providing conversion functions for 
serial to paralld and parallel to serial, said means 



15 



20 



25 



30 



35 



40 



45 



50 



55 



60 



65 



further comprising means to encode and decode header 
messages for a261, JPEG and MPEG; 
means for host processings said means providing a high 
speed interface between said communication assembly 
and an external host, said means furtho* providing a 
system independent communication between said con- 
troller assembly and said external host for the exdiangc 
sequence of command and data messages, said host 
means further providing windowing operations permit- 
ting the system memory to be memory mapped into a 
frame msatxy whereby said host processor can view 
firame memory status and operations in real time; 
means for system host bus, said bus means permitting said 
host i^ocessor to control access and communicate with 
network communication processors, transmission 
processors, pixd processors and frame memory; 
means for a communication ppelining subsystem, said 
means providing real time firame formatting, protocol 
controlling transmission and receiving; 
means for pixel processing, said means comprising a bus 
controller and frame memory controller wherein said 
pixd processing means serves as a bus master for a 
video bus wherein said pixel {n-ocessing means can 
access frame video memory for pixel levd operations, 
said means furtho- comprising Hi level manipulation 
capabilities for variable length coding and decoding, 
scan formatting conversion and quantization; 
means for display processing, said means for display 
processing performing digital YUV to digital RGB 
conversion, digital RGB to analog RGB convCTsion, 
digital RGB to VGA conversion and analog RGB to 
NTSC conversion; 
means for capture processing, said means for ca^>turc 
processing permitting decoding of analog video input 
fcffmats into COR 601, said means for capture pro- 
cessing further decoding said COR 601 into OF 
focrmulated YUV signals; 
means for frame storage, said means for frame storage 
con^vising a twin memory access storage for previous 
frames and current firames acccMtling to a formulated 
GIF format in order to facilitate frame <iifferencing 
operations; 

means fw motion i^^occssing, said means for motion 
processing induding a means for loading luminance 
signal in frame memory fOT pcrfcxming motion estima- 
tion and generating a motion vector for each block of 
storage in frame memory; 

means for transform processing, said means for transfc»m 
processing comparing the differences between the pre- 
vious and the present current blocks or maao blocks 
and coding said differences in discrete cosine transform 
cocffidents; 

means for video bus, said means for video bus providing 
a high speed, bi-directional, parallel interconnect 
between said frame memory, said capture processor, 
said disjrfay processor, said transfcam processor, said 
pixel processor and said motion processor, and 

means for video subsystem pipelining, said means com- 
prising system-wide direct interconnection between 
said capture processor, said pixel processor, said 
motion processor, said transform processor, said frame 
memory and said display processor. 
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